TECHNICAL  REPORT  2005-002 


QUALITY  EVALUATION  TOOL 
FOR  COMPUTER-  AND 
WEB-DELIVERED  INSTRUCTION 


JUNE  2005 


Robert  T.  Hays 

Naval  Air  Warfare  Center  Training  Systems  Division 
Ren6e  J.  Stout 
Renee  Stout,  Inc. 

David  L.  Ryan-Jones 

Naval  Air  Warfare  Center  Training  Systems  Division 


NAVAL  AIR  WARFARE  CENTER 
TRAINING  SYSTEMS  DIVISION 
ORLANDO,  FL  32826-3275 


APPROVED  FOR  PUBLIC  RELEASE 
DISTRIBUTION  IS  UNLIMITED 


D.  J.  DVVVER,  Head 
Human  Systems  Research 
&  Engineering  Division 


20050/11  065 


GOVERNMENT  RIGHTS  DATA  STATEMENT 


Reproduction  of  this  publication  in  whole  or  in 
part  is  permitted  for  any  purpose  of  the  United 
States  Government. 


Technical  Report  2005-002 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  t  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and  maintaining  the 
data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing 
this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management 
and  Budget.  Paperwork  Reduction  Project  (0704-0188),  Washington,  DC  20503. 

1.  AGENCY  USE  ONLY  (Leave  Blank) 

2.  REPORT  DATE 

June  2005 

3.  REPORT  TYPE  AND  DATES  COVERED 

Technical  Report 

May  2003  -  June  2005 

4.  TITLE  AND  SUBTITLE 

QUALITY  EVALUATION  TOOL  FOR  COMPUTER-  AND  WEB-DELIVERED 
INSTRUCTION  (U) 

5.  FUNDING  NUMBERS 

6.  AUTHOR(S) 

Robert  T.  Hays,  Renee  J.  Stout,  and  David  L.  Ryan-Jones 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Naval  Air  Warfare  Center  Training  Systems  Division 
(AIR  4652) 

12350  Research  Parkway 

Orlando,  FL  32826-3275 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 

Technical  Report 

2005-002 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Joint  ADL  Co-Laboratory  Navy  Human  Performance  Center 

13501  Ingenuity  Drive  Dam  Neck  Annex 

Suite  248  2025  Tartar  Ave. 

Orlando,  FL  32826  Virginia  Beach,  VA  23461 

10.  SPONSORING/MONITORING 
AGENCY  REPORT  NUMBER 

11.  SUPPLEMENTARY  NOTES 

12a.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 

12b.  DISTRIBUTION  CODE 

“A” 

13.  abstract  ( Maximum  200  words)  The  objective  of  this  effort  was  to  develop  an  Instructional  Quality  Evaluation  Tool  to  help 
instructional  developers  and  instructional  program  managers  quantitatively  determine  the  quality  of  their  instructional 
products,  both  during  development  and  at  project  completion.  The  Tool  can  also  support  the  comparison  of  courseware  from 
different  developers  or  courseware  that  uses  different  combinations  of  instructional  media.  Likert  scaling  techniques  were 
used  to  develop  5-point  rating  scales  for  each  evaluation  criterion.  Anchor  descriptions  were  developed  for  each  rating  point 
on  all  scales.  This  report  includes  these  anchored  Likert  scales,  which  can  serve  as  a  “stand-alone”  Tool.  The  evaluation 
criteria  are  organized  into  two  major  sections:  instructional  features  and  user-interface  design.  For  those  who  with  additional 
information  or  with  to  conduct  a  more  “in-depth”  evaluation,  the  detailed  discussions  of  each  evaluation  criterion  include 
definitions  of  important  terms  and  summaries  of  the  literature  that  supports  the  criterion.  The  discussions  also  provide 
suggested  techniques  to  improve  courseware  by  optimizing  the  instruction  from  the  perspective  of  each  criterion. 
Recommendations  for  future  improvement  of  the  Tool  are  provided. 

14.  SUBJECT  TERMS 

Instructional  Quality  Evaluation;  Instruction;  Learning;  Criterion-based  Evaluation 

15.  NUMBER  OF  PAGES 

176 

16.  PRICE  CODE 

17.  SECURITY  CLASSIFICATION 

OF  REPORT 

UNCLASSIFIED 

18.  SECURITY  CLASSIFICATION 

OF  THIS  PAGE 

UNCLASSIFIED 

1 9.  SECURITY  CLASSIFICATION 

OF  ABSTRACT 

UNCLASSIFIED 

>0.  LIMITATION  OF  ABSTRACT 

NSN  7540-0 1  -280-5500  Standard  Form  298  (Rev.  2-89) 

Prescribed  by  ANSI  Std  239-18 


298-102  i 


3 


Technical  Report  2005-002 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


4 


Technical  Report  2005-002 


EXECUTIVE  SUMMARY 


PROBLEM 

High  quality  instruction  is  essential  to  ensure  people  learn  the  knowledge  and  skills  that  will 
enable  them  to  succeed  at  their  jobs.  Computer-  and  web-delivered  instruction  has  the  potential 
to  provide  much  of  this  needed  instruction.  Although  the  military  and  industry  are  purchasing 
large  numbers  of  computer-based  and  web-delivered  courses,  there  is  currently  no  method, 
supported  by  empirical  and  theoretical  literature,  to  determine  if  this  courseware  is  of  good 
quality.  This  is  also  true  for  all  types  of  instruction.  However,  it  is  especially  critical  for 
computer-  and  web-delivered  courseware  because  these  often  do  not  include  an  instructor  to 
make  adjustments  to  the  instructional  content  “on  the  fly.” 

OBJECTIVE 

The  objective  of  this  effort  was  to  develop  a  tool  to  help  instructional  developers  and 
instructional  program  managers  determine  the  quality  of  their  instructional  products,  both  during 
development  and  at  project  completion.  The  tool  should  also  support  the  comparison  of 
courseware  from  different  developers  or  courseware  that  uses  different  combinations  of 
instructional  media.  It  should  be  designed  to  be  used  during  all  phases  of  instructional 
development  (i.e.,  support  both  formative  and  summative  evaluations).  Furthermore,  it  should 
be  designed  so  that  an  evaluator  has  the  choice  of  how  “deeply”  to  evaluate,  depending  on  his  or 
her  time  and  other  resources. 

APPROACH 

Based  on  the  results  of  earlier  work  (Hays,  Ryan-Jones,  &  Stout,  2003)  and  a  detailed  review 
of  the  literature  on  learning  and  instructional  evaluation,  a  series  of  evaluation  criteria  were 
developed.  Additional  literature  on  learning  and  instructional  theories  and  research  was 
reviewed  to  document  the  supporting  literature  for  the  evaluation  criteria.  Likert  scaling 
techniques  were  used  to  develop  5-point  rating  scales  for  each  evaluation  criterion.  Anchor 
descriptions  were  developed  for  each  rating  point  on  all  scales.  The  scales  were  designed  to 
“stand  alone”  as  the  evaluation  Tool.  The  detailed  discussions  were  designed  to  allow  evaluators 
to  obtain  additional  information  on  the  criteria  if  they  have  the  time  and  other  resources  to 
devote  to  a  more  “in-depth”  evaluation.  The  discussions  include  information  on  the  importance 
of  the  evaluation  criteria,  the  empirical  support  for  the  criteria,  and  suggestions  for  optimizing 
the  instructional  quality  based  on  the  criteria. 

FINDINGS 

The  Instructional  Quality  Evaluation  Tool  consists  of  a  series  of  criteria  (in  the  form  of 
anchored  Likert  scales)  for  evaluating  instructional  quality  and  directions  for  using  them.  The 
Tool  is  divided  into  two  main  sections:  instructional  features  evaluation,  which  includes  four 
evaluation  areas,  and  user-interface  design  evaluation,  which  includes  three  evaluation  areas. 
Each  evaluation  area  includes  several  evaluation  criteria.  The  scales  are  designed  to  “stand 
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alone”  as  the  evaluation  “Tool.”  An  evaluator  can  use  the  anchor  statements  and  instructions  to 
conduct  a  meaningful  evaluation  of  instructional  products  without  any  additional  information.  If 
an  evaluator  has  the  necessary  time  and  other  resources  to  conduct  a  more  in-depth  evaluation,  or 
if  he  or  she  wishes  additional  information  on  a  criterion,  he  or  she  may  consult  the  discussion 
sections.  The  detailed  discussions  of  each  evaluation  criterion  include  definitions  of  important 
terms  and  summaries  of  the  literature  that  supports  the  criterion.  The  discussions  also  provide 
suggested  techniques  to  improve  courseware  by  optimizing  the  instruction  from  the  perspective 
of  each  criterion. 

CONCLUSIONS  AND  RECOMMENDATIONS 

Instructional  developers  and  instructional  program  managers  can  use  the  Tool  during  all 
phases  of  instructional  product  development.  They  can  also  use  it  at  the  level  of  detail  that  their 
time  and  resources  permit.  During  product  development,  the  Tool  can  help  to  ensure  that  no 
important  facet  of  quality  is  overlooked.  At  project  completion,  the  Tool  can  help  to  determine 
if  the  instructional  product  includes  the  best  quality  courseware  for  their  investment. 
Recommendations  for  future  improvements  of  the  Tool  are  also  provided. 
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SECTION  I: 

A  TOOL  FOR  EVALUATING 
INSTRUCTIONAL  QUALITY 

This  report  is  the  first  version  of  the  Instructional  Quality  Evaluation  Tool 
(hereafter  called  the  Tool).  It  also  includes  summaries  of  the  supporting  literature 
upon  which  the  Tool  is  based  and  guidance  for  using  the  Tool. 


CHAPTER  Is 
INTRODUCTION: 

WHY  QUALITY  INSTRUCTION? 

Problem:  The  Need  for  Instructional  Quality 

Effective  learning  and,  ultimately,  improved  job  performance  are  the  goals  of  all  instructional 
systems.  In  recent  years,  many  organizations,  including  academia  and  those  in  the  public  and 
private  sectors,  have  turned  to  computer-based  instruction  and  distance  learning  (e.g.,  internet- 
delivered)  approaches  as  alternatives  to  traditional,  lecture-based  courses.  However,  the 
majority  of  instructors  surveyed  by  the  National  Education  Association  (over  50%)  believe  that 
their  traditional  course  in  the  same  subject  matter  does  a  better  job  in  meeting  educational  goals 
than  their  distance  learning  course  (National  Educational  Association,  2000).  This  document  is 
intended  to  help  improve  the  quality  of  distance  learning  and  other  computer-delivered 
instruction. 

In  order  to  help  ensure  that  all  instructional  products  meet  their  instructional  goals, 
instructional  developers  and  project  managers  need  assistance  in  evaluating  those  products 
during  development  and  at  the  time  of  delivery.  This  Tool  provides  a  method  to  help  them  in 
these  endeavors.  The  development  of  the  Tool  leveraged  earlier  work  on  guidelines  for 
designing  web-based  instruction  (Hamel,  Ryan-Jones,  &  Hays,  2000)  and  initial  instructional 
quality  evaluation  criteria,  in  the  form  of  a  checklist  (Hays,  Ryan-Jones,  and  Stout,  2003). 

Education  or  Training? 

A  distinction  is  sometimes  made  between  education  and  training.  Education  is  often  regarded 
as  the  teaching  of  general  knowledge,  while  training  is  believed  to  focus  on  the  teaching  of 
specific,  job-relevant  skills.  The  Tool  does  not  dwell  on  this  distinction  because  it  is  assumed 
that  both  require  instruction.  Instruction  is  the  process  by  which  individuals  learn  new 
knowledge  and  skills.  Therefore  learning  is  the  product  of  the  instructional  product.  If  we  want 
a  good  product,  we  need  to  ensure  that  the  process  is  working  as  effectively  as  possible.  The 
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instructional  quality  evaluation  criteria  apply  to  both  education  and  training.  In  discussions  of 
the  evaluation  criteria,  the  term  instruction  is  used  to  refer  to  both  education  and  training. 

What  is  Quality? 

Most  people  would  agree  that  quality  is  important  in  the  products  we  buy,  the  items  we 
create,  and  in  our  “quality  of  life.”  However,  less  agreement  is  found  when  we  try  to  define 
quality.  Hays  et  al.  (2003)  discuss  some  of  the  ways  quality  has  been  defined  and  conclude  that 
quality  is  found  in  the  interaction  of  persons  with  the  world  and  that  it  is  defined  by  the  standards 
they  set.  Botha  (2000)  elaborates  on  this  approach  by  describing  six  conceptions  of  quality,  as 
the  term  is  applied  to  instructional  products. 

•  Quality  as  “exceptional.”  In  this  sense,  saying  an  instructional  product  is  of  high  quality 
means  that  it  is  something  special,  distinctive,  of  high  class,  and  surpassing  very  high 
standards.  It  is  an  example  of  “excellence.” 

•  Quality  as  “perfection.”  This  conception  of  quality  means  that  there  are  minimal  defects 
(absence  of  defects)  in  an  instructional  product,  when  measured  against  a  set  of 
standards. 

•  Quality  as  “fitness  for  purpose.”  When  an  instructional  product  does  the  job  it  is 
designed  for  (e.g.,  it  meets  its  instructional  objectives),  it  can  be  said  to  be  of  high 
quality. 

•  Quality  as  “facilitating  changes”  in  learners.  When  a  learner  takes  a  course,  it  is  with  the 
intent  of  learning  new  knowledge,  skills,  and  abilities  (KSAs),  which  are  influenced  by 
the  learner’s  attitudes  (affect).  The  instructional  product  is  intended  to  transform  the 
learner.  If  these  changes  enable  the  learner  to  excel  in  his  or  her  chosen  field,  the  product 
can  be  said  to  be  of  high  quality. 

•  Quality  as  “value  for  money.”  Instructional  products  must  be  “purchased.”  If  the 
organization  that  procures  the  product  believes  they  have  received  good  value  for  their 
investment,  the  product  can  be  said  to  be  of  high  quality.  For  instructional  products 
designed  for  military  and  industrial  instructional  users,  the  most  unambiguous  method  to 
determine  good  value  is  demonstrated  improvement  in  job  performance  or  alternatively, 
equivalent  job  performance  for  lower  cost  instruction. 

Objective:  The  Purpose  of  the  Instructional  Quality  Evaluation  Tool 

As  Broadbent  and  Cotter  (2003)  observe,  the  word  evaluation  is  derived  from  the  word 
“value.”  A  quality  evaluation  seeks  to  establish  the  value  of  the  instructional  products  that  are 
developed  and  procured.  To  accomplish  this,  the  instructional  products  should  be  evaluated  on 
criteria  that  address  all  of  the  conceptions  of  quality  listed  above.  The  Tool  provides  a  set  of 
criteria  that  help  meet  this  objective.  These  evaluation  criteria  are  shown  in  Table  1,  which  also 
serves  as  a  summary  sheet  for  accumulating  evaluation  scores.  The  evaluation  criteria  are 
divided  into  two  main  sections:  instructional  features  criteria  and  user  interface  criteria. 

Although  many  of  the  criteria  apply  to  all  forms  of  instruction,  the  main  focus  of  the  method  is 
the  evaluation  of  computer-  and  web-delivered  instructional  products  that  are  designed  for  use 
without  the  presence  of  an  instructor. 
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Table  1: 

Instructional  Quality  Evaluation  Summary  Sheet 


Instructional  Features  Evaluation 

Rating  Criteria 

Score  (1-5)* 

1.  Instructional  Content 

1  .a.  The  content  is  presented  in  a  logical  manner. 

1  .b.  The  purpose  of  the  course  is  clearly  stated. 

1  .c.  The  instructional  objectives  are  clearly  stated. 

l.d.  The  content  supports  each  and  every  instructional  objective. 

l.e.  The  content  is  free  of  errors. 

V? ‘ 

1  .f.  The  content  is  job-relevant. 

1  .h.  There  are  clear  indications  of  prerequisites. 

•  -  • 

1  .i.  There  are  clear  indications  of  completed  topics. 

l.j.  Sources  for  additional  information  are  available. 

mm 

Content  Subtotal 

2.  Instructional  Activities 

'  '  •* 

2.a.  Activities  are  relevant  (all  support  LOs  &  job  requirements). 

2.b.  The  learner  is  required  to  interact  with  content. 

v>  .  '  ; 

2.C.  Instruction  is  engaging  (attracts  and  maintains  learners’  attention). 

i  ■ 

2.d.  Instructional  media  directly  support  learning  activities. 

h\.. 

Activities  Subtotal 

3.  Performance  Assessment 

3. a.  Assessments  are  relevant. 

UK 

3.b.  Assessments  are  logical. 

3.c.  Assessments  are  varied. 

HI 

Assessment  Subtotal 

4.  Performance  Feedback 

wm\ 

4.a.  Feedback  is  timely. 

MMM* 

mm\ 

4.b.  Feedback  is  meaningful  (related  to  objectives). 

4.c.  Positive  reinforcement  is  provided  for  correct  responses. 

4.d.  Remediation  is  provided  for  incorrect  responses. 

Feedback  Subtotal 

Instructional  Features  Subtotal  (carry  to  next  page) 

*Note:  No  matter  what  the  overall  score,  a  score  of  one  (1)  on  any  criterion  should  be  considered 
a  major  problem  and  requires  redesign  of  the  instructional  product.  A  score  of  two  (2)  on  any 
criterion  should  be  considered  a  problem  and  may  require  redesign. 
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Table  1: 

Instructional  Quality  Evaluation  Summary  Sheet 

(Continued) 


Score  (1-5)* 


User  Interface  Evaluation 


Evaluation  Criteria 


5.  Navigation  and  Operation 


5. a.  User  Interface  makes  course  structure  explicit 


5.b.  Tutorial  is  available  to  explain  navigation  and  operation  features. 


5.c.  Help  function  is  available  to  explain  navigation  &  operation  features. 


5.d.  Includes  all  necessary  navigation  and  operation  controls. 


5.e.  Navigation  &  operation  controls  are  clearly  and  consistently  labeled. 


5.f.  Navigation  &  operation  controls  are  located  in  consistent  place. 


.  Navigation  &  operation  controls  function  consistently. 


5.h.  Course  is  designed  to  show  learner’s  location. 


5.i.  Course  is  designed  to  show  how  learner  arrived  at  location. 


5.i.  Course  is  designed  to  show  estimated  time  required  for  each  module. 


Navigation  &  Operation  Subtotal 


6.  Content  Presentation 


6.a.  There  are  no  sensory  conflicts. 


6.b.  All  media  are  clear  and  sha 


6.c.  Presentations  are  aesthetically  pleasin 


6.d.  Multi-modal  presentation  of  content  is  used. 


6.e.  Multi-media  presentation  of  content  is  used. 


6.f.  Media  are  easy  to  use. 


;.  External  hyperlinks  are  kept  to  a  minimum. 


Presentation  Subtotal 


7.  Installation  and  Registration 


7.a.  Course  does  not  require  installation  or  learners  can  install  the  course 
without  assistance. 


7.b.  Minimal  “plug-ins”  are  required. 


7.c.  “Optimization”  test  is  available. 


7.d.  Technical  support  is  available. 


7.e.  Registration  is  simple  &  straightforward  (or  not  required 


Installation  and  Registration  Subtotal 


User  Interface  Subtotal  (from  this  page 


Instructional  Features  Subtotal  (from  previous  page 


Total  Quality  Score  (sum  of  subtotals 


*Note:  No  matter  what  the  overall  score,  a  score  of  one  (1)  on  any  criterion  should  be  considered 
a  major  problem  and  requires  redesign  of  the  instructional  product.  A  score  of  two  (2)  on  any 
criterion  should  be  considered  a  problem  and  may  require  redesign. 


H 


"■  •  .'i-' 


I 


ill 

ill 


14 


Technical  Report  2005-002 


Improvements  Included  in  the  Tool 

Hays,  et  al.  (2003)  recommended  two  improvements  and  expansions  of  the  original 
evaluation  checklist:  (1)  the  inclusion  of  detailed  rating  scales  for  each  criterion;  and  (2)  the 
development  of  discussions  of  each  criterion  so  that  evaluations  can  be  more  standardized  and 
made  easier  for  the  user.  This  report  meets  both  of  these  needs. 

The  scales  are  the  essence  of  the  Tool.  They  are  designed  to  support  an  evaluation  without 
the  need  to  consult  additional  information.  If  an  evaluator  has  limited  time  or  other  resources, 
the  scales  contain  all  of  the  necessary  information  to  conduct  a  meaningful  evaluation. 

However,  if  the  evaluator  has  enough  resources  to  conduct  a  more  “in-depth”  evaluation  or 
wishes  additional  information  on  the  importance  of  a  criterion  or  the  empirical  research  that 
supports  it,  he  or  she  may  consult  the  discussion  sections. 

The  discussions  are  intended  to  be  used  by  individuals,  like  those  on  instructional 
development  teams,  who  need  or  want  additional  details  about  the  evaluation  criteria.  These 
discussions  include  definitions  of  terms  and  recommendations  for  designing  instructional 
products.  The  discussions  also  include  summaries  of  the  empirical  and  theoretical  literature  that 
supports  each  criterion  and  suggestions  for  locating  resources  that  provide  additional  details. 

Such  detailed  discussions  are  necessary  because  evaluation  of  instructional  products  is  not  an 
easy  task.  Evaluation  usually  requires  the  evaluator  to  invest  time  and  attention  for  a  detailed 
review  of  a  course  or  other  instructional  product.  In  essence,  the  evaluator  may  have  to  “take  the 
course,”  often  multiple  times  from  different  perspectives  (e.g.,  a  novice  learner,  an  experienced 
learner,  an  instructor,  etc.).  An  example  of  the  recommended  steps  to  follow  in  evaluating 
courseware  is  shown  in  Table  2  (Coursestar,  2003).  This  is  a  good  beginning.  However,  the 
Tool  is  a  much  more  elaborate,  quantitative  method  that  allows  individuals  to  evaluate 
courseware  on  all  the  critical,  empirically-  and  theoretically-based  criteria  that  support  learning. 

Beyond  “buzz  words”.  Many  existing  evaluation  approaches  recommend  criteria,  such  as 
“the  content  has  educational  value,”  “presentation  of  content  is  clear  and  logical,”  “graphics  are 
appropriate,”  “package  is  motivational,”  or  “feedback  is  effectively  applied.”  However,  these 
criteria  are  seldom  explained,  nor  are  they  accompanied  by  some  form  or  “anchored”  evaluation 
scale.  Also,  the  terms  used  in  these  criteria  are  usually  left  to  the  user  to  define.  Without 
specific,  standard  definitions  of  terms  and  explanations  of  the  context  where  each  term  is 
applied,  evaluators  could  actually  be  evaluating  different  concepts.  Therefore,  this  Tool  includes 
“anchored”  evaluation  scales  and  the  report  also  includes  definitions  and  explanations  of  all 
terms  and  discussions  of  how  the  terms  are  used  in  the  context  of  the  evaluation  criteria. 

A  Systems  Approach.  An  instructional  system  (or  training  system)  is  the  planned  interaction 
of  people,  materials,  and  techniques,  which  has  the  goal  of  improved  performance  as  measured 
by  established  criteria  from  the  job  (Hays,  1992).  The  people  in  an  instructional  system  include, 
but  are  not  limited  to:  the  learner,  the  instructor,  the  course  developer(s),  effectiveness 
evaluators,  administrators,  logistics  managers,  subject  matter  experts,  instructional  media 
designers  and  implementers,  and  sometimes  on-the-job  supervisors.  Some  of  the  materials  in  an 
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instructional  system  are:  instructional  content,  instructional  media,  devices,  documents,  job- 
specific  equipment,  and  evaluation  instruments  (such  as  the  evaluation  scales  presented  in  this 
report).  Some  of  the  techniques  employed  in  instructional  systems  include:  needs  analysis, 
instructional  design  approaches,  instructional  aids  design  methods,  instructional  effectiveness 
evaluation  methods,  logistics  analysis,  and  cost-benefit  analysis  (often  called  return  on 
investment  analysis).  Finally,  some  of  the  interactions  which  take  place  in  instructional  systems 
include  those  during  development  and  evaluation.  They  also  include  instructor-learner 
interactions,  learner-content  interactions,  and  learner-learner  interactions  that  take  place  during 
instruction. 


Table  2: 

Recommended  Steps  to  Follow  In  Evaluating  Courseware 


Step  h  Run  the  program  briefly  to  become  familiar  with  its  operation  and  features. 
Step  2.  Execute  the  program  as  a  conscientious  learner,  avoiding  intentional  or  careless 
errors.  Extend  the  program  when  possible  by  interacting  as  a  creative  learner 
would. 


Step  3.  Execute  the  program  as  a  careless  learner  would.  Respond  incorrectly  to  test 
how  the  program  handles  learner  errors.  Repeat  the  same  response  and  also  try 
giving  different  incorrect  responses.  Make  other  kinds  of  errors  such  as  typing 
mistakes,  incorrect  form  of  input,  content  errors,  and  errors  in  following 
directions. 


Step  4.  Choose  a  checklist  for  the  evaluation  of  software  adapting  it  as  necessary  to  fit 
your  own  needs.  Key  factors  to  consider: 


Directions 


Input 


Screen  formatting 


Documentation 


The  directions  should  be  complete,  readable,  under  the 
user’s  control,  and  they  should  include  appropriate 
examples. 

A  program  should  ensure  that  a  user  knows  when  and  in 
what  form  input  is  needed.  Usually,  minimal  typing  should 
be  required. 

Are  the  screens  easy  to  understand-not  distracting  or 
cluttered? 

Includes  instructional  objectives,  suggested  activities, 
instruction  for  teacher  modification,  and  other  needed 
information? 


Step  5.  Record  your  evaluation  in  the  appropriate  form. 


Hays  discussed  a  variety  of  system  concepts  that  affect  the  quality  of  training  (or 
instructional)  systems  (1992)  and  web-based  instructional  products  (2001).  Foremost  among 
these  system  concepts  is  that  any  instructional  product  is  part  of  a  larger  instructional  system. 
As  a  system,  a  change  in  any  part  of  the  instructional  product  can  affect  the  system  as  a  whole. 
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It  is  assumed  that  lack  of  attention  to  quality  on  any  criterion  can  reduce  the  overall  quality  of 
the  instructional  product.  This  is  because  a  violation  of  any  criterion  could  become  a  distraction 
for  the  learners.  It  is  therefore  important  to  use  every  criterion  when  determining  overall  quality. 
Furthermore,  a  score  of  one  on  any  criterion  should  be  considered  to  be  a  serious  problem  and 
will  probably  require  some  level  of  redesign  of  the  product.  Further  explanations  are  provided  in 
the  section  on  interpretation  of  scores. 

Evaluating  instructional  products  is  a  difficult  and  complex  task  that  may  require  the 
evaluator  to  delve  into  a  course  with  almost  as  much  attention  as  a  learner.  For  developers, 
evaluation  of  their  instructional  products  should  be  an  ongoing  process  during  and  after 
development.  Resource  constraints  (e.g.,  time,  money)  can  limit  the  amount  of  effort  a 
developer  devotes  to  certain  elements  of  a  course.  Even  so,  it  cannot  be  emphasized  too  strongly 
that  each  and  every  one  of  the  criteria  in  the  Tool  should  be  evaluated  at  some  level.  In  addition, 
if  the  evaluator  finds  any  major  type  of  distraction  that  is  not  addressed  in  this  Tool,  he  or  she 
should  consider  revising  the  course. 

Organization  of  the  Report 

Chapter  2  presents  a  series  of  5-point  evaluation  scales — one  for  each  evaluation  criterion 
listed  in  Table  1.  All  the  scales  include  verbal  anchors  and  some  include  simple  examples.  It 
should  be  remembered  that  the  scales  are  designed  to  “stand  alone.”  Using  only  these  scales, 
evaluators  with  limited  time  or  resources  can  still  conduct  a  meaningful  evaluation.  After 
assigning  a  value  on  each  scale,  the  evaluator  can  then  transfer  the  scores  to  the  summary  sheets 
(Table  1)  to  generate  the  overall  evaluation  score. 

If  evaluators  are  able  to  devote  additional  effort,  they  can  “drill  down”  into  deeper  levels  of 
information  on  one  or  more  of  the  criteria  by  reading  the  discussions  in  Sections  II  and  III.  This 
is  strongly  recommended  for  instructional  developers.  These  discussions  provide  suggestions  on 
“how  to”  maximize  the  score  on  each  criterion.  They  also  include  summaries  of  the  literature 
that  supports  the  criterion.  A  series  of  appendices  supplements  the  discussions  with  additional 
information  on  some  of  the  more  complex  criteria.  Although  they  provide  substantial  details  on 
the  evaluation  criteria,  the  discussions  should  be  considered  as  overviews  on  the  topics.  Users  of 
the  Tool  who  wish  or  need  additional  details  should  consult  the  original  sources  cited  in  the 
discussions  and  listed  in  the  reference  section. 
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CHAPTER  2: 

THE  EVALUATION  SCALES 


The  following  scales  allow  an  evaluator  to  rate  computer-  and  web-delivered  instructional 
products  on  criteria  that  are  important  to  the  quality  of  instruction.  All  of  the  scales  are  scored 
on  a  5-point  scale  ranging  from  one  (1)  to  five  (5).  The  scales  include  descriptive  anchors  to 
help  the  evaluator  assign  a  score.  Some  of  the  scales  also  include  examples  or  important  things 
to  look  for  when  conducting  the  evaluation. 

To  assist  the  rater,  the  behavioral  anchor  statements  use  terms  like  “almost  none,”  “few,” 
“some,”  “most,”  and  “almost  all.”  A  few  of  the  scales  contain  anchor  statements  that  combine 
certain  features  (e.g.,  few  of  “this”  and  some  of  “that”).  It  should  be  noted  that  the  anchor  terms 
are  illustrative  and  should  only  be  used  as  a  guide  when  scoring.  Most  of  the  scales  ask  the  rater 
to  indicate  how  much,  how  often,  or  how  extensively  a  given  feature  appears  in  the  instructional 
product.  To  assist  in  choosing  a  rating,  it  may  be  helpful  to  think  in  terms  of  percentages.  Thus, 
a  rating  of  one  (1)  is  roughly  0-10%,  a  two  (2)  is  roughly  25%,  a  three  (3)  is  roughly  50%,  a  four 
(4)  is  roughly  75%,  and  a  five  (5)  is  roughly  90-100%.  Use  these  percentages  as  a  guide  in 
addition  to  the  anchor  statements  and  examples. 

The  scales  can  be  applied  by  an  individual  reviewer  or  they  might  be  used  in  a  group 
evaluation.  In  the  second  case,  the  ratings  on  each  criterion  could  be  an  average  of  all  the  ratings 
in  the  group.  Reiser  and  Kegelmann  (1994)  suggest  that  instructional  products  should  be 
evaluated  by  up  to  three  raters.  They  further  suggest  that  these  individuals  can  work  either 
independently  or  together  when  conducting  their  evaluations.  The  ratings  could  also  be 
consensus  scores,  assigned  after  group  discussions.  This  is  the  recommended  approach. 

Using  the  Scales  During  Different  Instructional  Development  Phases.  The  scales  can  be  used 
in  each  of  the  phases  of  instructional  development.  During  the  requirements  analysis  phase,  the 
scales  can  be  used  to  help  focus  the  analysis  to  identify  the  critical  content  information, 
presentation  alternatives,  and  implementation  issues.  An  important  part  of  this  analysis  is  to 
determine  the  prerequisites  and  purpose  of  the  course.  This,  in  turn,  will  help  determine  how  to 
structure  items  of  content  to  be  mutually  supportive. 

The  scales  can  be  applied  formatively  during  the  design  and  development  phase  to  help  the 
instructional  developers  focus  on  important  design  principles.  This  can  enable  the  developers  to 
recognize  and  correct  problems  before  they  are  “cast  in  stone.”  During  the  implementation 
phase,  the  scales  can  be  iteratively  used  to  help  ensure  that  implementation  issues  do  not  conflict 
with  or  override  important  instructional  requirements.  For  example,  a  course  that  requires  “plug¬ 
ins”  (extra  software  programs)  that  are  not  allowed  on  the  learner’s  computer  because  of 
organizational  policy  would  need  to  be  redesigned.  Finally,  during  the  evaluation  phase,  the 
scales  provide  the  capability  to  determine  strengths  of  the  course  and  also  weaknesses  that  may 
be  corrected  in  later  iterations. 
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Instructional  Features  Evaluation  Criteria 


The  criteria  in  this  section  are  used  to  evaluate  the  manner  in  which  the  instructional  content 
is  designed  and  organized  for  presentation  to  the  learner.  They  do  not  address  the  “appearance” 
and  presentation  of  the  content.  The  criteria  that  evaluate  these  issues  are  found  in  the  “User 
Interface”  section.  Additional  information  on  instructional  features  criteria  is  provided  in  the 
discussion  chapters  in  Section  II. 


1.  Instructional  Content 


The  first  evaluation  areas  consists  of  instructional  features  evaluation  criteria  that  deal  with 
how  the  content  is  organized  so  that  the  learner  receives  the  highest  quality  instruction  from  the 
course.  The  content  should  be  organized  and  presented  to  maximize  the  probability  that  learners 
will  learn  the  knowledge  and  skills  that  are  needed  from  the  course.  This  involves  motivating 
learners,  gaining  and  maintaining  their  attention,  and  helping  them  to  transfer  the  learned 
knowledge  and  skills  to  their  jobs.  The  instruction  should  be  designed  to  include  all  of  the 
necessary  events  for  effective  instruction  (e.g.,  Gagne  and  Briggs,  1979).  It  should  also  help 
learners  develop  confidence  that  they  are  obtaining  the  instruction  that  will  support  their  specific 
learning  needs. 

There  are  a  variety  of  labels  for  the  organizational  sections  of  a  course.  For  example,  one 
method  is  to  label  the  sections  as  follows:  Course,  Module,  Lesson,  Topic,  and  Subtopic.  In  this 
example,  each  section  following  “course”  is  a  smaller,  subset  of  the  section  preceding  it.  For  the 
purposes  of  this  Tool,  a  distinction  will  not  be  made  between  these  terms  because  there  is  so 
much  variation  in  how  they  are  used  in  the  instructional  development  community.  In  most 
sections  the  terms  “topic”  will  be  used. 
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l.a.  The  Content  is  Presented  in  a  Logical  Manner 


1 

2 

3 

4 

5 

Course  structure  is 

Course  structure  is 

Course  structure  is 

Course  structure  is 

Course  structure  is 

verv  illogical. 

illogical 

somewhat  logical 

logical. 

verv  logical. 

Instructions:  Using  the  anchor  statements  above  and  the  list  of  things  to  look  for  below, 
circle  the  number  that  best  describes  how  logically  the  course  content  is  presented. 

Things  to  look  for  to  determine  logical  presentation: 

•  Content  structure  is  clearly  visible  (e.g.,  table  of  contents,  course  outline;  see  also  5.a 
for  additional  details  on  how  the  user  interface  can  make  the  course  structure  explicit) 

•  Instructional  content  is  organized  to  support  learning  objectives  (see  l.c.) 

•  Learning  objectives  support  each  other  (also  see  Table  5) 

•  No  learning  objectives  conflict  (also  see  Table  5) 

•  Content  is  logically  consistent  (e.g.,  is  free  from  internal  conflicts) 

•  Instructional  content  builds  on  prior  learning 

•  Content  builds  in  complexity 

•  Content  uses  advance  organizers  and  summaries 

•  Content  appropriately  emphasizes  or  highlights  important  information  (e.g.,  uses  cues 
and  signals) 


(Please  go  the  to  next  page  for  Criterion  l.b) 
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l.b.  The  Purpose  of  the  Course  is  Clearly  Stated 


1 


2 


3 


4 


No  purpose  is 
stated. 


General  purpose 
statement  about 
course  or  general 
discussion  about 
learners  (e.g.,  who 
they  are),  but  not 
both. 


General  purpose 
statement  about 
course  and  general 
discussion  about 
learners  (e.g.,  who 
they  are). 


Specific  purpose 
statement  about 
course  and  general 
discussion  about 
learners  or  general 
purpose  statement 
about  course  mid 
specific  discussion 
about  learners 
(who  they  are,  why 
they  should  take 
the  course  and  the 
benefits  they  will 
gain). 


Specific  purpose 
statement  about 
course  and  specific 
discussion  about 
learners  (who  they 
are,  why  they 
should  take  the 
course  and  the 
benefits  they  will 
gain). 


Instructions:  Using  the  examples  below  and  anchor  statements  above,  circle  the  number  that 
best  characterizes  how  the  purpose  of  the  course  is  communicated  to  the  learner. 


Example  of  Increasing  Specificity,  Relevance,  and  Clarity  of  Purpose  Statements: 


(0)  N/A 


(1)  “The  purpose 
of  this  course  is 
to  teach  you  to 
be  a  better 
leader.” 

Or 

“This  course  is 
for  beginning 
supervisors.” 


(2)  “The  purpose 
of  this  course  is  to 
teach  beginning 
supervisors  how  to 
be  better  leaders.” 


(3)  “The  purpose 
of  this  course  is  to 
teach  beginning 
supervisors  how  to 
conduct  a 
performance 
appraisal  with  their 
subordinates.” 

Or 

“The  purpose  of 
this  course  is  to 
teach  beginning 
supervisors  to  be 
better  leaders  so 
they  can  better 
manage  their 
subordinates.” 


(4)  “Effective  performance 
appraisals  are  important 
because  they  help 
employees  know  how  they 
are  doing  on  their  jobs, 
what  they  need  to  do  to 
improve  their  performance, 
and  motivate  them  to 
achieve. 

The  purpose  of  this 
course  is  to  train  beginning 
supervisors  in  program 
management  to  conduct 
more  effective  performance 
appraisals  with  their 
subordinates. 

After  completing  this 
course  you,  a  beginning 
supervisor  in  program 
management,  will  be  better 
able  to  conduct 
performance  appraisals 
with  your  subordinates  that 
will  be  meaningful  and  will 
help  them  do  a  better  job.” 


(Please  go  to  the  next  page  for  Criterion  1  .c) 
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l.c.  The  Instructional  Objectives  Are  Clearly  Stated 


1  2  3  4  5 


No  objectives  are 
stated. 


General  objectives 
are  stated  at  the 
beginning  of  the 
course. 


General  objectives 
are  stated  at  the 
beginning  of  each 
course  topic. 


Objectives  are 
stated  at  the 
beginning  of  each 
course  topic  and 
include  brief 
statements 
describing  actions, 
standards,  & 
conditions. 


Detailed  objectives 
are  stated  at  the 
beginning  of  each 
course  topic  and 
include  detailed 
statements 
describing  all 
actions,  standards, 
&  conditions. 


Instructions:  Using  the  anchor  statements  above,  circle  the  number  that  indicates  how  clearly 
the  instructional  objectives  of  the  course  are  communicated  to  the  learner. 

Note:  If  you  give  a  rating  of  one  (1)  on  this  criterion,  you  must  also  rate  the  next  criterion  (l.d.) 
as  one  (1). 


l.d.  The  Content  Supports  Each  and  Every  Instructional  Objective 


1  2  3  4  5 

No  instructional  Few  instructional  Some  instructional  Most  instructional  All  instructional 

objectives  are  objectives  are  objectives  are  objectives  are  objectives  are 

supported.  supported.  supported.  supported.  supported. 

Instructions:  Using  the  anchor  statements  above,  circle  the  number  that  indicates  how  well 
the  course  content  supports  each  and  every  instructional  objective. 

Note:  If  the  score  on  l.c  is  one  (1)  you  must  also  give  this  criterion  a  score  of  one  (1). 


(Please  go  to  the  next  page  for  Criterion  1  .e) 


24 


Technical  Report  2005-002 


l.e.  The  Content  is  Free  of  Errors 


i 

Very  many  factual 
errors  and/or 
distraction  errors. 


2 

Many  factual 
errors  and/or 
distraction  errors. 


3 

Some  factual 
errors  and/or 
distraction  errors 


4 

Few  factual  errors 
and/or  distraction 
errors. 


5 

No  factual  errors 
or  distraction 
errors. 


Instructions:  Using  the  anchor  descriptions  above  and  the  list  of  types  of  errors  below,  circle 
the  number  that  best  indicates  how  much  the  course  is  free  of  errors. 

Types  of  Errors  and  Examples: 


Factual  Errors 

•  Incorrect  definitions 

•  Incorrect  formulas 

•  Math  errors 

•  Logical  errors  in  examples 
or  presentations 

•  Other  obvious  factual 
errors 


Distraction  Errors 

•  Typos 

•  Mislabeled  graphics 

•  Grammatical  errors 

•  Shifts  in  tense 

•  Shifts  in  voice  (e.g.,  human  to  computer¬ 
generated  or  changes  in  accent) 

•  Unexpected  addition  of  sounds  or  narration 

•  Shifts  in  sound  level 

•  Unexpected  changes  in  control  location  (see 
section  5.f.) 

•  Malfunctions  in  interface  controls  (see  section 

5-g.) 

•  Activities  that  do  not  perform  correctly  (e.g., 
drag  and  drop  answers  that  bounce  back  even 
when  correct) 

•  Other  obvious  distractions 


(Please  go  to  the  next  page  for  Criterion  l.f) 
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l.f.  The  Content  is  Job-Relevant 


1  2  3  4  5 

None  of  the  Little  of  the  Some  of  the  Most  of  the  AH  of  the  content 

content  is  job-  content  is  job  content  is  job  content  is  job  is  job  relevant, 

relevant  relevant.  relevant.  relevant. 

Instructions:  Using  the  statements  above  and  the  list  of  techniques  below,  circle  the  number 
which  best  indicates  the  job-relevance  of  the  course. 

Wavs  Content  Can  Be  Made  Job-Relevant: 

•  Includes  clear  statements  linking  information  to  job  tasks 

•  Examples  are  from  the  job  context 

•  Exercises  are  from  the  job 

•  Scenarios,  vignettes  are  from  the  job 

•  Includes  simulations  of  job  processes  and  equipment 

•  Shows  how  interconnections  among  items  support  job  tasks 

•  Includes  testimonials  and  stories  from  experts  at  the  job 

•  Provides  resources  for  future  job  support 


l.g.  The  “Authority”  for  the  Content  is  Clearly  Stated 


l 

•  No  references 

•  No  developer  or 
SME  credits 


2 

•  Only  a  general 
reference  section 
(e.g.,  specific 
“facts”  are  not 
referenced) 

•  No  developer  or 
SME  credits 


3 

•  Some  major 
“facts”  are 
referenced  & 
sources  are  listed 
in  a  reference 
section 

•  Some  references 
are  from  respected 
sources 

•  Provides  general 
credits  for 
developers  & 
SMEs  (names  or 
affiliations 


4 

•  Most  “facts”  are 
referenced  & 
sources  are  listed 
in  a  reference 
section 

•  Most  references 
are  from  respected 
sources 

•  Provides  general 
credits  for 
developers  & 
SMEs  (names  or 
affiliations) 


5 

•  Every  “fact”  is 
referenced  & 
sources  are  listed 
in  a  reference 
section 

•  AH  references 
are  from  respected 
sources 

•  Provides 
complete  credits 
for  developers  and 
SMEs  (names, 
affiliations,  and 
experience) 


Instructions:  Using  the  anchor  statements  above,  circle  the  number  which  best  characterizes 
the  level  of  “authority”  of  the  course  content. 

(Please  go  to  the  next  page  for  Criterion  1  .h) 
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l.h.  There  are  Clear  Indications  of  Prerequisites 


1  2  3  4  5 

No  indications  of  Few  needed  Some  needed  Most  needed  Every  needed 

needed  prerequisites  are  prerequisites  are  prerequisites  are  prerequisite  is 

prerequisites  clearly  indicated  clearly  indicated  clearly  indicated  clearly  indicated 

Instructions:  Using  the  anchor  statements  above  and  the  list  of  ways  to  indicate  prerequisites, 
circle  the  number  that  indicates  how  well  the  course  indicates  needed  prerequisites  for  the 
learner. 

Wavs  to  Indicate  Prerequisites: 

•  Includes  statements  of  any  required  prerequisite  course(s)  in  the  introduction 

•  Content  is  sequenced  from  simple  to  complex,  from  basic  to  detailed,  etc. 

•  Does  not  allow  learner  to  move  to  new  topic  until  previous  topic  has  been  completed  and 
the  learner  has  been  evaluated 


l.i.  There  Are  Clear  Indications  of  Completed  Topics 


1  2  3  4  5 

Always  unclear  Usually  unclear  Sometimes  clear  Usually  clear  when  Always  clear  when 

when  topics  are  when  topics  are  but  sometimes  topics  are  topics  are 

completed  completed  unclear  when  completed  completed 

topics  are 
completed 

Instructions:  Using  the  statements  above  and  the  examples  below  for  guidance,  circle  the 
number  that  indicates  how  clearly  the  course  indicates  completion  of  topics. 

Indications  that  a  Topic  Has  Been  Completed: 

•  Unambiguous  statements  that  topic  has  been  completed 

•  Includes  summaries  at  the  end  of  each  topic 

•  Includes  assessments  at  the  end  of  each  topic 

•  Includes  remediation  if  learner  has  not  mastered  each  topic 

•  Completed  topic  changes  color  in  the  table  of  contents 


(Please  go  to  the  next  page  for  Criterion  1  .j) 
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l.j.  Sources  for  Additional  Information  are  Available 

I _ I _ I _ I _ I 

1  2  3  4  5 


No  sources  are 
provided. 


A  brief 
bibliography  is 
provided  at  the  end 
of  the  course. 


A  detailed 
bibliography  is 
provided  at  the  end 
of  the  course. 


Each  course 
segment  includes  a 
section  containing 
sources  for 
additional 
information  and  a 
detailed 
bibliography  is 
provided  at  the  end 
of  the  course. 


Each  course 
segment  includes  a 
section  with 
sources  for 
additional 
information  and 
detailed 
instructions  for 
locating  and  using 
the  sources  for 
additional 
information  and  a 
detailed 
bibliography  is 
provided  at  the  end 
of  the  course. 


Instructions:  Using  the  statements  above,  circle  the  number  that  indicates  how  well  the 
course  provides  additional  information. 


(Please  go  to  the  next  page  for  Evaluation  Area  2) 
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2.  Instructional  Activities 


The  second  evaluation  area  includes  criteria  that  deal  with  the  instructional  activities  that  help 
the  learner  interact  with  the  content.  These  activities  are  essential  to  help  the  learner  process  the 
knowledge  and  skills  at  a  deeper  level  and  increase  the  probability  that  he  or  she  will  be  able  to 
perform  effectively  at  some  later  time  (e.g.,  on  the  job). 


2.a.  Activities  Are  Relevant 

(All  Support  Learning  Objectives  and  Job  Requirements) 


1 


2 


3 


4 


5 


None  of  the  Few  of  the 

activities  are  activities  are 

relevant  relevant. 


Some  of  the  Most  of  the  A]]  of  the  activities 

activities  are  activities  are  are  relevant, 

relevant.  relevant 


Instructions:  Using  the  anchor  statements  above  and  the  examples  below,  indicate  how  well 
the  course  provides  relevant  activities. 

Note:  If  you  rated  this  criterion  as  one  (1),  do  not  continue  ratings.  This  course  needs  major 
rework. 

Examples  of  Relevant  Activities 

•  Role  playing  in  a  case  study  based  on  real-world  issues 

•  Solving  realistic  (real-world)  problems 

•  Interacting  with  realistic  simulations 

•  Identifying  the  similarities  and  differences  among  real-world  examples 

•  Using  analogies 


(Please  go  to  the  next  page  for  Criterion  2.b) 
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2.b.  The  Learner  is  Required  to  Interact  with  Content 


1  2  3  4  5 


Learner  does  not 
interact  with 
content  or  only 
interacts  with  the 
interface  (e.g.,  to 
navigate) 


Learner  is  only 
required  to  answer 
simple  factual 
questions  before 
moving  to  a  new 
section. 


Learner  is  required 
to  answer  simple 
factual  questions 
and  engage  in 
simple  activities 
like  “drag  and 
drop”  before 
moving  to  a  new 
section. 


Learner  is  required 
to  answer  factual 
questions  and 
engage  in  a  variety 
of  activities 
ranging  from 
simple  to  complex 
before  moving  to  a 
new  section. 


Learner  is  required 
to  answer  factual 
questions  and 
engage  in  a  wide 
variety  of  activities 
ranging  from 
simple  to  complex 
before  moving  to  a 
new  section. 

the  degree  that  the 


Instructions:  Using  the  statements  above,  circle  the  number  that  indicates 
learner  is  “required”  to  interact  with  the  content  of  the  course. 


2.c.  Instruction  is  Engaging 
(Attracts  &  Maintains  Learners’  Attention) 


l 


2 


3 


5 


None  of  the 
content  is 
engaging. 


Little  of  the 
content  is 
engaging. 


Some  of  the 
content  is 
engaging. 


Most  of  the  Al]  of  the  content 

content  is  is  engaging, 

engaging. 


Instructions:  Based  upon  your  experience  with  the  course  and  the  examples  below,  circle  the 
number  that  indicates  whether  you  agree  or  disagree  that  the  instruction  is  engaging. 

Examples: 

•  Session  length  is  not  too  long 

•  Gains  and  maintains  learner  attention  through  appearance  of  content 

•  Content  is  challenging  (e.g.,  builds  from  simple  to  complex;  uses  competition) 

•  Shows  learner  that  he  or  she  is  attaining  course  objectives 

•  Uses  expert  testimonials  and  real-world  examples  to  make  instruction  more  relevant. 

•  Engages  learner’s  curiosity  and  fantasy 

•  Uses  variety  in  media  and  instructional  methods 

(Please  go  to  the  next  page  for  Criterion  2.d) 
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2.d.  Instructional  Media  Directly  Support  Learning  Activities 


12  3  45 


None  of  the  media 
support  learning 
activities.  All  are 
gratuitous. 


A  few  of  the  media 
directly  support 
learning  activities. 
Most  are 
gratuitous. 


Some  of  the  media 
directly  support 
learning  activities. 
Some  are 
gratuitous. 


Most  of  the  media 
directly  support 
learning  activities. 
A  few  are 
gratuitous. 


All  media  directly 
support  learning 
activities.  None 
are  gratuitous. 


Instructions:  Using  the  anchor  statements  above,  circle  the  number  that  indicates  how  much 
the  instructional  media  support  the  learning  activities  in  the  course. 

Note:  The  term  gratuitous  refers  to  the  use  of  media  that  are  included  only  because  they  add 
glamour  or  “flash”  to  the  course,  but  do  not  add  to  the  instructional  value  of  the  course  (e.g.,  they 
do  not  explain,  elaborate,  or  clarify). 


(Please  go  to  the  next  page  for  Evaluation  Area  3) 
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3.  Performance  Assessment 


The  third  evaluation  area  includes  criteria  that  are  used  to  determine  how  well  the 
instructional  product  assesses  whether  the  learners  are  learning  the  material. 


3.a.  Assessments  Are  Relevant 


1 

2 

3 

4 

5 

None  of  the 

Few  of  the 

Some  of  the 

Most  of  the 

All  of  the 

assessments  are 

assessments  are 

assessments  are 

assessments  are 

assessments  are 

relevant. 

relevant. 

relevant. 

relevant. 

relevant. 

Instructions:  Below  is  a  list  of  some  characteristics  of  relevant  assessments.  Read  the  list 
and,  using  the  anchor  statements  above,  circle  the  number  that  indicates  how  many  of  the 
assessments  are  relevant. 

Note:  If  you  assign  a  score  of  one  (1)  on  this  criterion,  you  must  also  assign  a  score  of  one  (1)  to 
the  next  two  criteria  (3.b.  &  3.C.). 

Characteristics  of  Relevant  Assessments: 

•  Related  to  topics  and  issues  that  the  learner  cares  about  (e.g.,  will  help  on  job) 

•  Learner  must  demonstrate  knowledge  or  skill  (“do”  versus  “know”) 

•  Assessment  activities  taken  from  or  apply  to  job  requirements 


(Please  go  to  the  next  page  for  Criterion  3.b) 
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3.b.  Assessments  Are  Logical 


1  2  3  4  5 

No  assessments  Few  assessments  Some  assessments  Most  assessments  All  assessments 

are  logical.  are  logical.  are  logical.  are  logical.  are  logical. 

Instructions:  Using  the  list  of  features  of  logical  assessments  and  the  anchor  statements 
above,  circle  the  number  that  indicates  how  many  of  the  assessments  are  logical.  In  general,  an 
assessment  that  has  more  of  the  features  would  indicate  a  more  logical  assessment. 

Note:  If  you  assigned  a  score  of  one  (1)  to  Criterion  3. a.,  you  must  also  assign  a  score  of  one  (1) 
to  this  criterion. 

Features  of  Logical  Assessments: 

•  Cogently  support  Learning  Objectives 

•  Are  iterative  (assess  progress  from  less  to  more  complex  information  or  tasks) 

•  Learner  expectations  are  managed  by  clearly  linking  assessments  to  purpose  and 
objectives  of  course 


3.c.  Assessments  Are  Varied 


1 

2 

3 

4 

5 

No  assessments 
are  used. 

Onlv  one  tvoe  of 
assessment  is  used. 

Little  variety  in 
assessments  is 
used  (they  are 
mostly  the  same). 

Some  variety  in 
assessments  is 
used  (there  are 
some  different 
types). 

Much  variety  in 
assessments  is 
used  (there  are 
several  different 
types). 

Instructions:  Using  the  anchor  statements  above  and  the  examples  below,  circle  the  number 
that  best  describes  how  much  variety  is  used  in  assessment  methods. 

Note:  If  you  assigned  a  score  of  one  (1)  to  Criterion  3. a.,  you  must  also  assign  a  score  of  one  (1) 
to  this  criterion. 


Example  Assessments: 

•  Multiple  choice  questions  on  factual  information 

•  Fill  in  the  blank  statements  of  factual  information 

•  True/false  statements  of  factual  information 

•  Completing  puzzles  based  on  correct  answers 


•  Drag  and  drop  to  match  questions  and  answers 

•  Role  playing  of  real-world  situations 

•  Simulations  of  real-world  problems 

•  Assessment  games  (must  include  measures  of 
performance) 


(Please  go  to  the  next  page  for  Evaluation  Area  4) 
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4.  Performance  Feedback 

Feedback  is  an  essential  component  of  high  quality  instruction.  If  well  designed,  feedback 
serves  as  another  instructional  opportunity.  The  fourth  evaluation  area  includes  criteria  that 
address  how  the  instructional  product  provides  feedback  to  the  learner  about  how  well  they  are 
learning  the  material. 

4. a.  Feedback  is  Timely 


1  2  3  4  5 

No  feedback  is  Feedback  only  at  Feedback  at  end  of  Feedback  at  end  Feedback  at  end  of 

provided  the  end  of  the  each  major  every  segment.  each  activity 

course.  segment.  within  a  segment. 


Instructions:  Using  the  anchor  statements  above,  circle  the  number  which  indicates  the 
timeliness  of  the  feedback  provided  in  the  course. 

Note:  If  you  give  a  one  (1)  rating  on  this  criterion,  you  must  also  rate  the  next  three  criteria 
(4.b.,  4.c.,  &  4.d.)  as  one  (1). 


4.b.  Feedback  is  Meaningful 


1 

2 

3 

4 

5 

No  feedback  is 

Little  of  the 

Some  of  the 

Most  of  the 

All  of  the  feedback 

provided 

feedback  is 
meaningful. 

feedback  is 
meaningful. 

feedback  is 
meaningful. 

is  meaningful. 

Instructions:  Using  the  statements  above  and  the  examples  below  circle  the  number  that 
indicates  how  much  meaningful  feedback  is  provided  to  the  learner. 

Note:  If  you  rated  the  previous  criterion  (4.a)  as  one  (1),  you  must  also  rate  this  criterion  as  one 

(1). 

Examples  of  meaningful  feedback: 

•  Is  based  on  concrete  learning  goals  (instructional  objectives)  that  are  clearly  understood 
by  the  learners 
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•  Describes  the  gap  between  the  learner’s  learning  goal  and  the  learner’s  current 
performance  and  suggests  how  to  close  the  gap 

•  Focuses  the  learner’s  attention  on  the  learning  goal  and  not  on  failure  to  achieve  the  goal 

•  Is  appropriate  for  the  ability  level  of  the  learner  (e.g.,  not  too  complex  for  novices; 
detailed  enough  for  experts) 

•  Provides  additional  information 

•  Corrects  learner  errors 

•  Asks  for  clarification 

•  Debates  issues  and  explains  disagreements 

•  Ties  concepts  to  one  another 

•  Offers  insight  based  on  expert  experience 

4.c.  Positive  Reinforcement  is  Provided  for  Correct  Responses 


1 

No  positive 
reinforcement  is 
provided. 


2 

Positive 

reinforcement  is 
usually  not 
provided  for 
correct  responses. 


3 

Positive 

reinforcement  is 
sometimes 
provided  for 
correct  responses. 


4 

Positive 

reinforcement  is 
usually  provided 
for  correct 
responses. 


5 

Positive 

reinforcement  is 
always  provided 
for  correct 
responses. 


Instructions:  Using  the  anchor  statements  above  and  the  definition  of  positive  reinforcement 
below,  circle  the  number  that  indicates  how  often  positive  reinforcement  is  provided  for 
incorrect  responses. 

Note:  If  you  rated  Criterion  4.a  as  one  (1),  you  must  also  rate  this  criterion  as  one  (1).  Also  note, 
in  an  instructional  context,  reinforcement  should  be  more  than  simply  indicating  that  the  answer 
is  “correct.” 

Definition  of  Positive  Reinforcement: 

Positive  reinforcement  is  a  stimulus  (e.g.,  praise,  food,  or  money)  provided  after  someone 
emits  a  behavior  that  increases  the  probability  that  the  behavior  will  occur  again.  Positive 
reinforcement  can  be  given  in  many  ways  and  under  many  different  schedules  (e.g.,  after  every 
behavior  [continuous  reinforcement],  after  some  number  of  behaviors  [fixed-ratio  schedule], 
after  a  set  time  period  [fixed-interval  schedule]).  Specific  types  of  reinforcement  and 
reinforcement  schedules  must  be  determined  in  the  context  of  the  course.  Ideally,  reinforcement 
should  be  another  instructional  experience. 


(Please  go  to  the  next  page  for  Criterion  4.d) 
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4.d.  Remediation  is  Provided  for  Incorrect  Responses 


1 

2 

3 

4 

5 

No  remediation  is 

Remediation  is 

Remediation  is 

Remediation  is 

Remediation  is 

provided. 

provided  for  few 

provided  for  some 

provided  for  most 

provided  for  all 

incorrect 

incorrect 

incorrect 

incorrect 

responses. 

responses. 

responses. 

responses. 

Instructions:  Using  the  examples  below  and  the  anchor  statements  above,  circle  the  number 
that  indicates  how  often  remediation  is  provided  for  incorrect  responses. 

Note:  If  you  rated  Criterion  4.a  as  one  (1),  you  must  also  rate  this  criterion  as  one  (1). 

Examples  of  Remediation: 

•  Explains  why  the  response  was  incorrect 

•  Provided  in  a  non-threatening,  encouraging  manner 

•  Provides  the  correct  answer  or  demonstrates  correct  skill  performance 

•  Encourages  the  learner  to  obtain  more  instruction  on  correct  behavior 

•  Leads  the  learner  to  remedial  instruction 

•  Also  see  Criterion  4.b. 


(Please  go  to  the  next  page  for  User  Interface  Evaluation  Criteria  [Evaluation  Areas  5-7]) 
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User  Interface  Evaluation  Criteria 


The  User  Interface  criteria  deal  with  how  the  instructional  content  is  presented  to  the  learner, 
how  the  learner  moves  through  the  content,  and  how  he  or  she  controls  instructional  activities 
(e.g.,  playing  a  video).  In  general,  the  user  interface  should  be  designed  to  make  things  easier 
for  the  learner  (it  should  be  “learner-centric”).  Additional  information  on  these  criteria  is 
provided  in  the  discussion  chapters  in  Section  III. 

5.  Navigation  and  Operation 

This  evaluation  area  includes  criteria  about  how  the  learner  is  shown  the  course  structure  and 
provided  the  means  to  move  about  the  course  and  operate  the  controls  for  course  activities. 


5.a.  User  Interface  Makes  Course  Structure  Explicit 


1 

2 

3 

4 

5 

No  course 

Basic  course 

Basic  course 

Detailed  course 

Detailed  course 

structure  is  shown 

structure  is  shown 

structure  is  shown 

outline 

outline 

on  user  interface. 

on  the  user 

on  the  user 

(expandable  as 

(expandable  as 

interface  (e.g., 

interface  (e.g., 

needed)  is  shown 

needed)  is  shown 

course  outline), 

course  outline). 

on  the  interface. 

on  the  interface. 

but  does  help  the 

Summaries  or 

Summaries  or 

Summaries  or 

learner  understand 

other  methods  are 

other  methods  are 

other  methods  are 

how  course 

used  to  help  the 

used  to  help  the 

used  to  help  the 

segments  fit 

learner  understand 

learner  understand 

learner  understand 

together  and 

only  how  the 

how  all  course 

how  all  course 

support  one 

major  course 

segments  fit 

segments  fit 

another. 

segments  fit 
together  and 
support  one 
another. 

together  and 
support  one 
another. 

together  and 
support  one 
another.  When 
instructionally 
usefiil,  the  GUI 
allows  learner  to 
easily  move  to 
different  segments 
of  the  course. 

Instructions:  Using  the  anchor  statements  above  and  the  examples  below  circle  the  number 
that  best  describes  the  user  interface  of  the  course. 

Example  wavs  to  make  the  course  structure  explicit: 

•  Interface  includes  the  course  structure  as  part  of  its  main  display  and  indicates  the 
learner’s  current  location  at  all  times. 
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•  Course  outline  (table  of  contents)  always  available  and  can  be  expanded  to  show 
detailed  level 

•  Course  map  shows  course  structure  and  segments  are  accessible  from  course  map 

•  Summaries  of  course  segments  are  available  as  “previews.” 

•  Shows  history  of  course  segments  the  learner  has  completed 

•  Search  function  is  available  to  locate  information  on  specific  topics 


5.b.  Tutorial  is  Available  to  Explain 
Navigation  &  Operation  Features 


1 

2 

3 

4 

5 

No  tutorial  is 

Tutorial  explains 

Tutorial  explains 

Tutorial  explains 

Tutorial  explains 

provided. 

only  a  few  of  the 

some  or  the 

most  of  the 

all  of  the 

navigation  & 

navigation  & 

navigation  & 

navigation  & 

~mr  .  .  m  _  _ 

operation  features. 

operation  features. 

operation  features. 

operation  features. 

Instructions:  Using  the  anchor  statements  above  and  the  examples  below  circle  the  number 
that  best  describes  the  tutorial  provided  with  the  course. 


Wavs  to  design  tutorials: 

•  Walk-through  with  text  and  narration 

•  Animation  of  moving  cursor  with  text  narration 

•  Animated  “guide”  (avatar) 

•  Text  and  graphics 


(Please  go  to  the  next  page  for  Criterion  5.c) 
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5.c.  Help  Function  is  Available  to  Explain 
Navigation  &  Operation  Features 


1 

No  help  function  is 
provided. 


2 

Help  function  is 
sometimes 
available,  but  it  is 
often  difficult  to 
get  to  help 
function  and  find 
help  on  a  specific 
feature. 


3 

Help  function  is 
always  available, 
but  it  is  sometimes 
difficult  to  get  to 
help  function  and 
find  help  on  a 
specific  feature. 


4 

Help  is  always 
available  and 
easily  accessed 
from  any  part  of 
the  course.  Help 
on  a  specific 
feature  is  usually 
available. 


5 

Help  is  always 
available  and 
easily  accessed 
from  any  part  of 
the  course.  Help 
on  a  specific 
feature  is  always 
available  and  is 
easy  to  find. 


Instructions:  Using  the  anchor  statements  above  and  the  examples  below  circle  the  number 
that  best  describes  the  help  functions  available  in  the  course. 

Examples  of  wavs  to  access  help: 

•  Right-click  on  a  control  to  access  text  and  narration 

•  Roll  over  and  pause  on  control  to  access  text  and  narration 

•  Click  on  help  button  to  ask  question  or  access  alphabetical  list  of  instructions  or 
explanations 

•  Access  to  specific  segments  of  the  tutorial  to  explain  features 


(Please  go  to  the  next  page  for  Criterion  5.d) 
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5.d.  Includes  All  Necessary  Navigation  &  Operation  Controls 


1 

2 

3 

4 

5 

No  interaction 

Includes  few 

Includes  some 

Includes  most 

Includes  a]l 

after  course  starts. 

navigation 

navigation 

navigation 

navigation 

Course  leads 

controls.  Little 

controls.  Some 

controls. 

controls.  Learner 

learner  through  the 
material  at  a 
predetermined 
pace. 

learner  control. 

learner  control. 

Considerable 
learner  control  of 
media  and  time  on 
topic. 

has  complete 
control  of 
activities  and 
media,  time  on 
topic,  and 
movement  (except 
for  sequence  of 
instructional 
segments). 

Instructions:  Using  the  anchor  statements  above  and  the  list  of  possible  navigation  and 
operation  controls  below,  circle  the  number  that  indicates  if  the  course  includes  all  navigational 
controls  needed  to  move  through  the  course  and  effectively  interact  with  the  content  (e.g., 
operate  presentation  media  like  video  players). 

Note:  Only  consider  the  navigation  and  operation  controls  that  are  necessary  in  the  course  (e.g., 
if  no  videos  are  includes  then  no  VCR-type  controls  are  needed  and  you  should  not  lower  the 
score). 


Possible  Navigation  and  Operation  Controls: 


Start 

Stop 

Pause 

Resume  (may  use 
“start”  button) 
Mark  place  in 
course  (bookmark) 
VCR-type  controls 
for  videos  (start, 
stop,  pause,  rewind, 
fast  forward,  etc.) 


Go  forward  by 
screen 

Go  backward  by 
screen 

Go  forward  by 
lesson 

Go  backward  by 
lesson 

Return  to  main 
menu 


Return  to  beginning 
of  course 
Print  a  screen 
Print  selected  text 
Take  notes 
Save  a  copy  of  work 
Volume  control  for 
audio 

Search  function 
(e.g.,  keywords) 


(Please  go  to  the  next  page  for  Criterion  5.e) 
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5.e.  Navigation  &  Operation  Controls  are  Clearly  &  Consistently 

Labeled 


1 

2 

3 

4 

5 

None  of  the 

Few  of  the 

Some  of  the 

Most  of  the 

All  of  the 

navigation  & 

navigation  & 

navigation  & 

navigation  & 

navigation  & 

operation  controls 

operation  controls 

operation  controls 

operation  controls 

operation  controls 

are  clearly  and 

are  clearly  and 

are  clearly  and 

are  clearly  and 

are  clearly  and 

consistently 

consistently 

consistently 

consistently 

consistently 

labeled. 

labeled. 

labeled. 

labeled. 

labeled. 

Instructions:  Using  the  anchor  statements  above,  circle  the  number  that  indicates  how  clearly 
and  consistently  the  navigation  and  operation  controls  are  labeled  throughout  the  course. 
Minimally,  your  opinion  should  be  based  on  observing  the  navigation  controls  in  segments  at  the 
beginning,  middle,  and  end  of  the  course  and  also  observing  more  than  one  screen  within  each 
segment.  It  is  always  better  to  observe  more  of  the  course  before  giving  a  score. 


5f.  Navigation  &  Operation  Controls  are  Located  in  a  Consistent  Place 


1  2  3  4  5 


Almost  no  controls 
are  located  in  the 
same  place 
throughout  the 
course. 


Few  controls  are 
located  in  the  same 
place  through  the 
course. 


Some  controls  are 
located  in  the  same 
place  throughout 
the  course. 


Most  controls  are 
located  in  the  same 
place  throughout 
the  course. 


All  controls  are 
located  in  the  same 
place  throughout 
the  course. 


Instructions:  Using  the  anchor  statements  above  circle  the  number  that  best  indicates  how 
many  navigation  and  operation  controls  are  consistently  located.  Minimally,  your  rating  should 
be  based  on  observing  the  navigation  controls  in  segments  at  the  beginning,  middle,  and  end  of 
the  course  and  also  observing  more  than  one  screen  within  each  segment.  It  is  always  better  to 
observe  more  of  the  course  before  giving  a  score. 


(Please  go  to  the  next  page  for  Criterion  5.g) 
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5.g.  Navigation  &  Operation  Controls  Operate  Consistently 


1 

2 

3 

4 

5 

None  of  the 

Few  of  the 

Some  of  the 

Most  of  the 

All  of  the  controls 

controls  operate 

controls  operate 

controls  operate 

controls  operate 

operate 

consistently. 

consistently. 

consistently. 

consistently. 

consistently. 

Instructions:  Using  the  anchor  statements  above,  circle  the  number  that  best  characterizes 
how  consistently  the  navigation  and  operation  controls  operate  throughout  the  course. 
Minimally,  your  opinion  should  be  based  on  using  the  navigation  and  operation  controls  in 
segments  at  the  beginning,  middle,  and  end  of  the  course  and  also  trying  them  on  more  than  one 
screen  within  each  segment.  It  is  always  better  to  try  the  controls  in  more  of  the  course  before 
assigning  a  score. 


5.h.  Course  is  Designed  to  Show  Learner’s  Location 


1  2  3  4  5 

Very  difficult  to  Difficult  to  Somewhat  difficult  Easy  to  determine  Very  easy  to 

determine  location  determine  location  to  determine  location  in  course.  determine  location 

in  course.  in  course.  location  in  course.  is  course 

Instructions:  Using  the  anchor  statements  above  and  the  examples  below,  circle  the  number 
that  indicates  how  easy  it  is  for  the  learner  to  know  his  or  her  location  in  the  course. 

Example  Techniques  to  Help  Learner  Determine  Their  Location  in  the  Course: 

•  Course  map  (outline)  always  available  with  current  location  highlighted 

•  Descriptive  labels  for  current  topic  always  available  on  interface 


(Please  go  to  the  next  page  for  Criterion  5.i) 
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5.i.  Course  is  Designed  to  Show  How  Learner  Arrived  at  Location 


1  2  3  4  5 


Very  difficult  to 
determine  how  one 
arrived  at  a 
location  in  the 
course. 


Difficult  to 
determine  how  one 
arrived  at  a 
location  in  the 
course. 


Somewhat  difficult 
to  determine  how 
one  arrived  at  a 
location  in  the 
course. 


Easy  to  determine 
how  one  arrived  at 
a  location  in  the 
course. 


Very  easy  to 
determine  how  one 
arrived  at  a 
location  in  the 
course. 


Instructions:  Using  the  anchor  statements  above  and  the  examples  below,  circle  the  number 
that  indicates  how  easy  it  is  for  the  learner  to  know  how  he  or  she  arrived  at  a  location  in  the 
course. 


Example  Techniques  to  Help  Learner  Determine  How  They  Arrived  at  a  Course  Location: 

•  Course  map  showing  “trail”  learner  has  taken  through  course 

•  Highlighted  “completed”  topics  on  course  outline 

•  Prerequisites  must  be  completed  before  course  “allows”  learner  to  begin  a  segment 


5.j.  Course  is  Designed  to  Show  Estimated  Time 
Required  for  Each  Module 


1  2  3  4  5 

Time  estimates  Time  estimates  are  Time  estimates  are  Time  estimates  are  Time  estimates  are 

never  are  provided  seldom  provided.  sometimes  usually  provided.  always  provided. 

provided. 

Instructions:  Using  the  anchor  statements  above  and  the  examples  below,  circle  the  number 
that  indicates  how  often  the  course  provides  estimated  time  to  complete  segments  or  activities.  It 
is  suggested  that  at  least  three  segments  (topics  or  subtopics)  be  reviewed  to  make  this 
determination  (more  is  always  better). 

Example  Techniques: 

•  Simple  statement  (e.g.,  “this  module  should  take  about  10  minutes”) 

•  Numerical  display  (e.g.,  completed:  3  minutes  of  10  minutes) 

•  Countdown  clock 

•  Sliding  scale 


(Please  go  to  the  next  page  for  Evaluation  Area  6) 
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6.  Content  Presentation 


The  sixth  evaluation  area  includes  criteria  that  deal  with  how  the  content  is  presented.  This 
primarily  refers  to  the  appearance  or  multi-media  presentation  of  the  content. 


6.a.  There  Are  No  Sensory  Conflicts 


1 

Different  content 
is  always 
presented 
simultaneously  in 
two  or  more 
modalities. 


2 

Different  content 
is  often  presented 
simultaneously  in 
two  or  more 
modalities. 


3 

Different  content 
is  sometimes 
presented 
simultaneously  in 
two  or  more 
modalities. 


4 

Different  content 
is  seldom 
presented 
simultaneously  in 
two  or  more 
modalities. 


5 

Different  content 
is  never  presented 
simultaneously  in 
two  or  more 
modalities. 


Instructions:  Using  the  anchor  statements  above  and  the  examples  below  indicate  how  often 
the  course  contains  sensory  conflicts. 


Examples  of  sensorv  conflicts: 

Media  Mix 

Example 

Text  and  Audio 

Text  provides  different  information  from 
that  being  presented  on  an  audio  track  at 
the  same  time. 

Text  and  Video 

Text  provides  different  information  from 
that  being  presented  in  a  video  being 
played  at  the  same  time. 

Text  and 

Similar  to  the  scrolling  text  used  in  some 

Conflicting  text 

TV  news  broadcasts.  Learner  can’t  attend 
to  both  sources. 

Video  and  Audio 

Audio  track  is  not  synchronized  with  video 
or  is  providing  different  information. 

Sidebars 

Extra  textual  information  is  provided  in  a 
box  (sidebar),  but  the  learner  does  not 
know  when  to  read  this  extra  information 
or  why  it  is  provided. 

(Please  go  to  the  next  page  for  Criterion  6.b) 
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6.b.  All  media  are  Clear  and  Sharp 


1 

2 

3 

4 

5 

None  of  the  media 

Few  of  the  media 

Some  of  the  media 

Most  of  the  media 

All  of  the  media 

are  clear  and 

are  clear  and 

are  clear  and 

are  clear  and 

are  clear  and 

sharp. 

sharp. 

sharp. 

sharp. 

sharp. 

Instructions:  Using  the  examples  below  and  the  anchor  statements  above,  circle  the  number 
that  indicates  how  many  of  the  media  in  the  course  are  clear  and  sharp. 


Examples  of  Clear  and  Sharp  Media: 

•  Text:  Clear  and  sharp  text  is  readable  for  most  people  without  causing  eye  strain.  The 
specific  font  size  necessary  for  readability  depends  on  the  medium  (e.g.,  smaller  text  is 
easier  to  read  on  paper).  Larger  text  is  always  easier  to  read. 

•  Graphics:  Clear  and  sharp  graphics  are  large  enough  that  all  important  components  can 
be  distinguished  and  any  text  is  readable. 

•  Audio:  Clear  and  sharp  audio  is  loud  enough  that  all  words  and  important  sounds  (e.g., 
equipment  sounds)  can  easily  be  heard  and  understood. 

•  Video:  Clear  and  sharp  video  is  focused  on  important  components  and  uses  long  shots 
and  close-ups  to  establish  the  context  for  those  components. 


6.c.  Screens  are  Aesthetically  Pleasing 


12  3  45 


None  of  the 
screens  are 
aesthetically 
pleasing. 


Few  of  the  screens  Some  of  the 

are  aesthetically  screens  are 

pleasing.  aesthetically 

pleasing. 


Most  of  the  AH  of  the  screens 

screens  are  are  aesthetically 

aesthetically  pleasing, 

pleasing. 


Instructions:  Using  the  examples  below  and  vour  experience  of  the  course,  circle  the  number 
that  indicates  how  many  of  the  screens  are  aesthetically  pleasing. 

Examples:  All  aesthetic  experience  is  personal.  However,  artists  use  certain  principles  that  result 
in  positive  aesthetic  experiences  for  most  people.  Some  of  these  principles  are: 

•  simplicity 

•  balance  or  symmetry 

•  logical  groupings  of  items  by  proximity  and/or  similarity 
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•  colors  do  not  clash 

•  colors  are  used  to  convey  important  information 

•  white  space  is  used  to  separate  different  ideas  and  highlight  important  information 


6.d.  Multi-modal  Presentation  of  Content  is  Used 


1  2  3  4  5 


Multiple 

presentation  modes 
are  randomly  used 
with  no 
instructional 
purpose. 


Multiple 

presentation  modes 
are  seldom  used  to 
reinforce  and 
clarify 
instructional 
information. 


Multiple 

presentation  modes 
are  sometimes 
used  to  reinforce 
and  clarify 
instructional 
information. 


Multiple 

presentation  modes 
are  often  used  to 
reinforce  and 
clarify 
instructional 
information. 


Multiple 

presentation  modes 
are  always  used  to 
reinforce  and 
clarify 
instructional 
information. 


Instructions:  Using  the  anchor  statements  above  and  the  examples  below,  circle  the  number 
that  indicates  how  often  multi-modal  presentation  is  used  in  the  course. 


Examples  of  multi-modal  presentation: 

•  An  animation  is  used  to  clarify  a  textual  description  of  the  movement  of  oil  in  an 
engine 

•  A  video  shows  the  correct  placement  of  text  equipment  on  an  electrical  panel 

•  Important  points  in  an  audio  presentation  are  emphasized  with  bulleted  text 


(Please  go  to  the  next  page  for  Criterion  6.e) 
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6.e.  Multi-media  Presentation  of  Content  is  Used 


1 

2 

3 

4 

5 

Only  one 

Text  and  narration 

Text  and  narration 

Multiple  media 

Multiple  media 

presentation 
medium  is  used 
(e.g.,  text). 

with  still  graphics. 

with  still  graphics, 
and  videos.  Some 
gratuitous  use  of 
media. 

(text,  videos, 
animations,  still 
graphics)  are  used 
to  present  content, 
but  do  not 
consistently 
reinforce  each 
other.  No 
gratuitous  use  of 
media. 

(text,  videos, 
animations,  still 
graphics)  are  used 
to  present,  clarify, 
and  reinforce 
instructional 
information.  No 
gratuitous  use  of 
media. 

Instructions:  Using  the  statements  above  and  the  examples  below  circle  the  number  that 
indicates  how  multiple  media  are  used  to  present  course  content.  Note:  The  term  “gratuitous” 
means  the  inclusion  of  media  for  no  instructional  purpose,  but  only  to  add  “glitz”  to  the  course. 


Examples  of  Instructional  Media: 

•  Classroom  lecture  (or  talking  head) 

•  Video  tele-training  lecture 

•  Audio  lecture 

•  Video  lecture 

•  Text  instruction 

•  Animations 

•  Instructional  Games 

•  Job  Performance  Aid  (JPA) 


(Please  go  to  the  next  page  for  Criterion  6.f) 
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6.f.  Media  Are  Easy  to  Use 


1 

2 

3 

4 

5 

Media  are  never 

Media  are  seldom 

Media  are 

Media  are  usually 

Media  are  always 

easy  to  use. 

easy  to  use. 

sometimes  easy  to 

use. 

easy  to  use. 

easy  to  use. 

Instructions:  Using  the  examples  below,  circle  the  number  that  indicates  how  many  of  the 
media  are  easy  to  use. 


Examples  of  Easy-to-use  Media: 

•  Standard  browser  buttons 

•  Single-click  functions 

•  Familiar  VCR-type  video  controls 

•  Slide  control  for  audio  volume 


6.g.  External  Hyperlinks  are  kept  to  a  Minimum 


1  2  3  4  5 

Very  many  Many  external  Some  hyperlinks  Few  external  No  external 

external  hyperlinks  hyperlinks  are  are  used.  hyperlinks  are  hyperlinks  are 

are  used.  used.  used.  used. 


Instructions:  Using  the  statements  above,  circle  the  number  that  indicates  how  many  external 
hyperlinks  are  used  in  the  course.  External  hyperlinks  take  the  learner  to  an  external  site  that  is 
not  a  part  of  the  course  itself.  The  content  of  the  external  site  cannot  be  controlled  or  the  site 
may  no  longer  exist  at  the  time  the  learner  takes  the  course.  These  are  distinguished  from 
internal  hyperlinks  that  allow  the  learner  to  move  within  the  course  and  are  not  addressed  in  this 
criterion. 


(Please  go  to  the  next  page  for  Evaluation  Area  7) 
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7.  Installation  and  Registration 

The  seventh  evaluation  area  includes  criteria  that  deal  with  how  the  learner  installs  the  course 
to  run  on  his  or  her  computer. 


7.a.  Course  Does  Not  Require  Installation  or 
Learners  Can  Install  the  Course  Without  Assistance 


1  2  3  4  5 

Course  requires  Course  requires  Course  requires  Course  requires  Course  does  not 

installation  and  it  installation  and  it  installation  and  it  installation  and  it  require  installation 

is  very  difficult.  is  difficult.  is  somewhat  easy.  is  easy.  or  it  is  very  easy  to 

install. 


Instructions:  Using  the  statements  above  as  a  guide,  circle  the  number  that  indicates  how 
difficult  it  is  to  install  the  course. 


7.b.  Minimal  “Plug-ins”  Are  Required 


1  2  3  4  5 

Very  many  plug-  Many  plug-ins  are  Some  plug-ins  are  Few  plug-ins  are  No  plug-ins  are 
ins  are  required.  required.  required.  required.  required. 

Instructions:  Using  the  statements  above  as  a  guide,  circle  the  number  that  indicates  how 
many  plug-ins  are  required  to  run  the  course. 

Definition  of  “plug-ins”:  A  plug-in  is  a  program  needed  to  run  a  portion  of  a  course.  For 
example,  a  video  program  may  be  needed  to  play  the  videos  in  a  course.  Sometimes  plug-ins 
may  be  automatically  downloaded  from  the  internet. 


(Please  go  to  the  next  page  for  Criterion  7.c) 
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7.c.  “Optimization”  Test  is  Available 


2 


3 


4 


5 


No  “optimization” 
test. 


“Optimization” 
test  provides  tryout 
of  a  few  features, 
but  no  directions 
for  setting  up  the 
computer  to  best 
view  the  course. 


“Optimization” 
test  provides  tryout 
of  most  features 
but  no  directions 
for  setting  up  the 
computer  to  best 
view  the  course. 


“Optimization” 
test  provides  tryout 
of  most  features 
and  directions  for 
setting  up  the 
computer  to  best 
view  the  course. 


“Optimization” 
test  provides  tryout 
of  ah  features  and 
directions  for 
setting  up  the 
computer  to  best 
view  the  course. 


Instructions;  Using  the  anchor  statements  above,  circle  the  number  that  indicates  the  quality 
of  the  optimization  test. 


Definition  of  an  “Optimization”  Test: 

An  “optimization”  test  is  a  software-based  routine  that  helps  the  learner  try  out  all  of  the 
features  of  the  course  to  ensure  that  his  or  her  computer  is  set  up  to  support  the  optimal 
presentation  of  all  media  used  in  the  course. 


7.d.  Technical  Support  is  Available 


1 

2 

3 

4 

5 

No  technical 

Almost  no 

Minimal  technical 

Adeauate  technical 

Hiehlv  resoonsive 

support  is 

technical  support 

support  available, 

support  available. 

and  effective 

available. 

available. 

but  difficult  to 
obtain. 

technical  support 
available. 

Instructions:  Using  the  anchor  statements  above  and  the  characteristics  listed  below,  circle 
the  number  that  indicates  the  quality  of  technical  support  provided  with  the  course. 

Characteristics  of  Effective  Technical  Support: 

•  Easy  to  obtain 

•  Responsive  to  needs 

•  Available  any  time 

(Please  go  to  the  next  page  for  Criterion  7.e) 
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7.e.  Registration  is  Simple  and  Straightforward 
(Or  Not  Required) 


1 

2 

3 

4 

5 

Registration  is 

Registration  is 

Registration  is 

Registration  is 

Registration  is 

very  difficult. 

difficult. 

sometimes 

difficult. 

easy. 

very  easy  or  not 
reauired. 

Instructions:  Using  the  statements  above,  circle  the  number  that  indicates  how  easy  it  is  to 
register  for  the  course. 

Note:  If  registration  is  not  required,  assign  a  score  of  five  (5)  this  criterion. 
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INTERPRETATION  OF  SCORES 


Once  the  evaluator  has  assigned  a  score  on  the  criteria  in  all  seven  evaluation  areas,  he  or  she 
should  transfer  the  scores  to  the  summary  sheets  (Table  1)  and  add  them  to  generate  an  overall 
score  for  the  instructional  product.  The  following  is  a  suggested  scheme  for  the  interpretation  of 
the  scores. 


There  are  43  evaluation  criteria.  The  highest  possible  score:  215. 


Range  of  Scores 

Interpretation 

172-215 

Extremely  well  designed  instructional  product.  Scored  well  on 
all  criteria. 

151-171 

Has  potential.  Scored  well  on  all  “essential”  criteria,  but  still 
has  some  “loose  ends”  that  could  be  improved. 

129-150 

Has  some  strengths,  but  large  deficits.  Should  focus  on 
improving  weaknesses. 

108-128 

Not  enough  effort  invested  in  instructional  design.  Confusing 
and  could  lead  to  frustration.  Should  go  back  to  the  “drawing 
board.” 

43-107 

Inadequate.  Little  or  no  consideration  of  instructional  design. 

Not  suitable  for  most  learners. 

*Note:  No  matter  what  the  overall  evaluation  score,  a  score  of  one  (1)  on  any  criterion  should  be 
considered  a  major  problem  that  will  require  redesign  of  the  instructional  product.  A  score  of 
two  (2)  on  any  criterion  should  be  considered  a  problem  and  may  require  redesign. 


Interpretation  of  Scores  for  Each  Evaluation  Area: 

Looking  at  the  subtotal  scores  for  various  evaluation  areas  can  help  diagnose  specific 
problems  in  the  course.  The  table  on  the  next  page  provides  guidance  for  interpretation  of 
subtotal  scores.  If  a  problem  is  found  in  an  evaluation  area,  the  ratings  for  one  or  more  specific 
criteria  can  provide  more  detailed  diagnosis. 
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Evaluation  Areas 

Interpretation  of  Subtotal  Scores 

Instructional  Features 

105  is  the  highest  possible  subtotal  score  for  all 
instructional  features  criteria.  A  subtotal  score 
below  63  is  a  problem. 

Instructional  Content 
(Evaluation  Area  1) 

50  is  the  highest  possible  subtotal  for 
instructional  content  criteria.  A  subtotal  score 
below  30  is  a  problem. 

Instructional  Activities 
(Evaluation  Area  2) 

20  is  the  highest  possible  subtotal  score  for 
instructional  activities  criteria.  A  subtotal  score 
below  12  is  a  problem. 

Performance  Assessment 
(Evaluation  Area  3) 

15  is  the  highest  possible  subtotal  score  for 
performance  assessment  criteria.  A  subtotal 
score  below  9  is  a  problem. 

Performance  Feedback 
(Evaluation  Area  4) 

20  is  the  highest  possible  subtotal  score  for 
performance  feedback  criteria.  A  subtotal 
score  below  12  is  a  problem. 

User  Interface 

1 10  is  the  highest  possible  subtotal  score  for  all 
user  interface  criteria.  A  subtotal  score  below 

66  is  a  problem.  (See  note  under  Area  7, 
below) 

Navigation  and  Operation 
(Evaluation  Area  5) 

50  is  the  highest  possible  subtotal  score  for 
navigation  criteria.  A  subtotal  score  below  30 
is  a  problem. 

Presentation 
(Evaluation  Area  6) 

35  is  the  highest  possible  subtotal  score  for 
presentation  criteria.  A  subtotal  score  below 

2 1  is  a  problem. 

Installation  &  Registration 
(Evaluation  Area  7) 

25  is  the  highest  possible  subtotal  score  for 
installation  and  registration  criteria.  A  subtotal 
score  below  1 5  is  a  problem. 

54 


Technical  Report  2005-002 


SECTION  II: 

DETAILED  DISCUSSIONS  OF 
INSTRUCTIONAL  FEATURES  EVALUATION 

CRITERIA 


The  following  discussions  provide  information  to  help  the  evaluator  understand  and  more 
confidently  assign  ratings  to  each  of  the  instructional  features  evaluation  criteria.  Each  chapter 
in  this  section  presents  discussions  for  an  instructional  features  evaluation  area  (e.g.,  Area  1: 
Instructional  Content;  Area  2:  Instructional  Activities;  etc  ).  The  discussions  begin  with 
directions  for  using  the  scales.  This  is  followed  by  definitions  of  terms  and  discussions  of 
concepts  and  techniques  to  maximize  the  quality  of  instructional  products  as  measured  by  each 
criterion.  The  discussions  also  provide  summaries  of  some  of  the  literature  that  supports  each 
criterion. 

Learner-Centric  Evaluations.  In  conducting  the  evaluation,  the  evaluator  should  try  to  think 
in  terms  of  how  the  learner  will  perceive  the  instructional  product.  Thus  the  evaluation  should 
be  “learner-centric.”  For  example,  if  the  evaluator  is  trying  to  determine  if  the  purpose  of  the 
course  is  clearly  stated  (criteria  1  .b.),  he  or  she  should  ask  whether  the  learner  can  understand  the 
purpose  of  the  course.  Although  it  is  vitally  important  for  the  developer  to  understand  the 
purpose  of  the  course,  this  understanding  is  moot  if  it  is  not  communicated  to  the  learner. 

It  is  often  assumed  that  any  source  of  information  (e.g.,  a  web  site)  is  also  instruction  (e.g.,  a 
web  course).  This  is  because  many  do  not  understand  the  distinction  between  information  and 
instruction.  Winner  (as  cited  in  Iseke-Bames,  1996)  maintains  that  only  when  information  is 
interpreted  and  related  through  critical  thought  and  understanding  does  it  become  knowledge. 
People  can  learn  from  information,  but  it  is  uncontrolled  and  unstructured  learning  that  may 
produce  unpredictable  results.  Instruction  is  the  controlled  process  of  helping  the  learner  convert 
information  into  knowledge  (in  its  widest  sense,  this  also  includes  skills  or  abilities).  The 
instructional  process  is  also  affected  by  the  learner’s  attitudes  (affect).  Apart  from  factors 
outside  our  control,  instruction,  if  well  designed  and  implemented,  gives  us  confidence  that  the 
learner  has  learned  what  is  required  from  a  given  instructional  product. 
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CHAPTER  3: 

INSTRUCTIONAL  CONTENT 
(EVALUATION  AREA  1) 

The  first  evaluation  area  deals  with  the  manner  in  which  instructional  content  is  designed  and 
organized  for  presentation  to  the  learner.  The  “appearance”  of  the  content  is  discussed  in  the 
“User  Interface  Evaluation”  chapters  in  Section  III.  The  instructional  features  criteria  discussed 
below  apply  to  all  instructional  theories  or  approaches,  although  specific  tasks  may  lend 
themselves  to  different  instructional  approaches  or  combinations  of  approaches  (e.g., 
“behavioral,”  “cognitive,”  “constructivist,”  etc.).  The  focus  of  these  criteria  is  how  they 
contribute  to  the  goal  of  converting  information  into  knowledge. 


l.a.  The  Content  is  Presented  in  a  Logical  Manner 

This  criterion  is  used  to  evaluate  the  structure  of  the  content  and  how  it  is  designed  to 
facilitate  understanding  of  the  material  to  be  learned.  This  may  be  the  most  important  of  all  the 
instructional  evaluation  criteria.  If  the  content  is  illogically  structured,  learners  are  less  likely  to 
form  the  enduring  associations  among  separate  items  of  information  that  will  help  them 
remember  and  use  the  content  on  the  job.  Even  worse,  they  may  form  incorrect  associations  or 
patterns  among  items  of  information  that  could  lead  to  poor  performance. 

Using  the  Scale 

This  scale  rates  the  course  in  terms  of  how  logically  the  instructional  information  is  presented 
and  how  the  individual  pieces  of  information  are  related  to  one  another.  Evaluators  should  use 
the  anchor  statements  and  the  “things  to  look  for  to  determine  logical  presentation”  to  assign  a 
score  on  this  criterion.  Additional  information  to  help  raters  understand  the  terms  and  concepts 
involved  in  this  criterion  and  the  supporting  literature  for  this  criterion  is  provided  in  the 
following  sections. 

Terms,  Concepts,  Methods,  and  Supporting  Literature 

Researchers  have  studied  how  individuals  form  mental  patterns  (models)  of  information 
items.  This  research  has  shown  that  people  search  for  consistency  in  the  information  they 
encounter  in  all  areas  of  their  lives.  According  to  Johnson-Laird,  Legrenzi,  and  Girotto  (2004), 
“Individuals  evaluate  the  consistency  of  a  set  of  propositions  by  searching  for  a  model  of  a 
possibility  in  which  they  are  all  true.  If  they  find  such  a  model,  they  evaluate  the  propositions  as 
consistent;  otherwise,  they  evaluate  the  propositions  as  inconsistent”  (p.  42).  This  is  also  true  for 
information  that  learners  encounter  in  instructional  situations. 

A  problem  arises  when  an  individual  is  not  able  to  construct  a  model  that  provides 
consistency  among  pieces  of  information.  Because  they  strive  for  such  consistency,  they  may 
“succumb  to  illusions  of  consistency  and  of  inconsistency”  (Johnson-Laird,  et  al.,  2004,  p.  44). 


57 


Technical  Report  2005-002 


The  result  of  improperly  formed  models  of  instructional  information  is  likely  to  be  poorer 
performance.  Thus,  the  purpose  of  high  quality  instructional  content  should  be  to  guide  the 
learner  to  form  models,  patterns,  and  associations  of  information  that  will  be  more  easily 
retrieved  and  used  when  needed. 

This  criterion  evaluates  the  logical  structure  of  the  course  content.  It  focuses  on  whether 
instructional  topics  are  presented  in  a  logical  sequence  that  helps  the  learner  understand  how 
individual  topics  should  fit  together  to  support  improved  performance. 

Course  content  should  be  designed  to  follow  a  logical  structure  that  supports  student  learning. 
According  to  Merrill,  “associative  memory  is  composed  of  a  hierarchical  network  structure” 
(1983,  p.  302).  Any  information  to  be  learned  can  be  designed  so  that  its  structure  supports  the 
development  of  associative  memory.  Rosch  (1978)  believes  that  information  can  be  organized 
into  “basic”  concepts  and  then  into  more  specific  (subordinate)  or  more  general  (super-ordinate) 
categories.  Associative  memory  can  be  aided  if  the  information  is  clearly  structured  to  support 
the  interrelationships  among  these  levels  of  specificity. 

It  is  recommended  that  developers  ensure  that  the  content  includes  all  of  the  necessary  events 
of  instruction.  Although  various  instructional  event  models  have  been  developed,  the  most 
comprehensive  is  probably  the  nine  instructional  event  model  developed  by  Gagne  and  his 
associates  (e.g.,  Gagne  &  Briggs,  1979). 

The  Nine  Events  of  Instruction.  Although  differing  in  specifics,  most  instructional  models 
address  some  of  Gagne’s  nine  events  of  instruction  (Gagne  &  Briggs,  1979,  p.  166).  For 
example,  Reigeluth  and  Stein  (1983)  focus  on  how  to  structure  the  sequence  and  presentation  of 
the  ideas  in  instructional  content.  Keller  (1983)  provides  guidance  on  how  to  motivate  the 
learner.  Gagne’s  model  provides  a  comprehensive  overview  of  the  entire  instructional  process 
and  can  be  made  more  specific  by  applying  one  or  more  of  these  other  models.  The  nine 
instructional  events  and  the  conditions  of  learning  they  imply  for  five  types  of  learned 
capabilities  are  summarized  in  Table  3.  As  a  minimum,  the  content  should  be  designed  to 
include  all  nine  of  the  instructional  events  combined  with  the  appropriate  instructional  strategies. 

Instructional  Strategies.  Merrill  (1997)  describes  instructional  strategies  as  consisting  of  five 
parts.  He  states  that: 

A  complete  instructional  strategy  consists  of  knowledge  structure  consistent 
with,  and  appropriate  for,  the  knowledge  and  skill  being  taught,  a  presentation 
consistent  with,  and  appropriate  for  the  kind  of  knowledge  or  skill  being  taught, 
an  opportunity  for  exploration  of  the  ideas  being  taught,  practice  with  feedback 
consistent  with,  and  appropriate  for,  the  knowledge  or  skill  being  taught,  and 
learner  guidance  consistent  with,  and  appropriate  for,  the  knowledge  and  skill 
being  taught  (Merrill,  1997,  online  version,  p.  2,  author’s  emphasis). 


58 


Technical  Report  2005-002 


Table  3: 

Nine  Instructional  Events  and  Conditions  of  Learning 

(Adapted  from  Gagne  &  Briggs,  1979,  p.  166) 


Instructional  Event 

Capability  Type  and  Implications 

1.  Gaining  attention:  Present 
a  new  problem  or  situation. 

•  For  all  tvnes  of  capabilities:  Hein  the  learner  selectively  attend  to  the 
important  information  and  stimuli  in  the  task  to  be  learned.  Use  “interest 
devices”  or  short  “teasers”  to  grab  learner’s  attention  (e.g.,  storytelling, 
demonstrations,  incorrect  performance  and  its  consequences,  why  it  is 
important). 

2.  Informing  the  learner  of 
the  objective:  Allow  the  learner 
to  organize  their  thoughts  around 
what  they  are  about  to  see,  hear, 
and/or  do. 

•  Intellectual  skill:  Provide  a  description  and  example  of  the  performance 
to  be  expected. 

•  Cognitive  strategy:  Clarify  the  general  nature  of  the  solution  exoected. 

•  Information:  Indicate  the  kind  of  verbal  question  to  be  answered. 

•  Attitude:  Provide  example  of  the  kind  of  action  choice  aimed  for. 

•  Motor  skill:  Provide  a  demonstration  of  the  performance  to  be  expected. 

3.  Stimulating  recall  of 
prerequisites:  Allow  the 
learner  to  build  on  previous 
knowledge  or  skills. 

•  Intellectual  skill:  Stimulate  recall  of  subordinate  concepts  and  rules. 

•  Cognitive  strategy:  Stimulate  recall  of  task  strategies  and  associated 
intellectual  skills. 

•  Information:  Stimulate  recall  of  context  of  organized  information. 

•  Attitude:  Stimulate  recall  of  relevant  information,  skills,  and  human 
model  identification. 

•  Motor  skill:  Stimulate  recall  of  executive  sub-routine  and  part-skills. 

4.  Presenting  the  stimulus 
material:  Chunk  the  information 
to  avoid  memory  overload. 

Structure  material  following 
instructional  strategies  (e.g., 

Merrill  (1997;  see  Instructional 
Strategies  Section). 

•  Intellectual  skill:  Present  examples  of  concent  or  rule. 

•  Cognitive  strategy:  Present  novel  problems. 

•  Information:  Present  information  in  propositional  form. 

•  Attitude:  Present  human  model  demonstrating  choice  of  personal  action. 

•  Motor  skill:  Provide  external  stimuli  for  performance,  including  tools  or 
implements. 

5.  Providing  learning 
guidance:  Provide  instructions 
on  how  to  learn. 

•  Intellectual  skill:  Provide  verbal  cues  to  proper  combining  sequence. 

•  Cognitive  strategy:  Provide  prompts  and  hints  to  novel  solution. 

•  Information:  Provide  verbal  links  to  a  larger  meaningful  context. 

•  Attitude:  Provide  for  observation  of  model’s  choice  of  action,  and  of 
reinforcement  received  by  the  model. 

•  Motor  skill:  Provide  practice  with  feedback  of  performance 
achievement. 

6.  Eliciting  the 
performance:  Provide 
opportunity  to  practice. 

•  Intellectual  skill:  Ask  learner  to  applv  rule  or  concept  to  new  examples. 

•  Cognitive  strategy:  Ask  for  problem  solution. 

•  Information:  Ask  for  information  in  paraphrase,  or  in  learner’s  own 
words. 

•  Attitude:  Ask  learner  to  indicate  choices  of  action  in  real  or  simulated 
situations. 

•  Motor  skill:  Ask  for  execution  of  the  performance. 
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Table  3: 

(Continued) 


Instructional  Event 

Capability  Type  and  Implications 

7.  Providing  feedback: 

Analyze  the  learner’s  behavior  and 
provide  them  with  specific 
feedback  on  degree  of  correctness, 
reinforcement  for  correct 
responses,  or  remediation  for 
incorrect  responses. 

•  Intellectual  skill:  Confirm  correctness  of  rule  or  concent  aonlication. 

•  Cognitive  strateev:  Confirm  originality  of  problem  solution. 

•  Information:  Confirm  correctness  of  statement  of  information. 

•  Attitude:  Provide  direct  or  vicarious  reinforcement  of  action  choice. 

•  Motor  skill:  Provide  feedback  on  degree  of  accuracy  and  timing  of 
performance. 

8.  Assessing  performance: 

Test  to  determine  if  the  lesson  has 
been  learned. 

•  Intellectual  skill:  Learner  demonstrates  aDDlication  of  concent  or  rule. 

•  Cognitive  strategy:  Learner  originates  a  novel  solution. 

•  Information:  Learner  restates  information  in  paraphrased  form. 

•  Attitude:  Learner  makes  desired  choice  of  personal  action  in  real  or 
simulated  situation. 

•  Motor  skill:  Learner  executes  performance  of  total  skill. 

9.  Enhancing  retention  and 
transfer:  Help  the  learner  use 
what  has  been  learned  (e.g., 
additional  practice,  similar 
problem  situations,  reviews). 

•  Intellectual  skill:  Provide  spaced  reviews  including  a  variety  of 
examples. 

•  Cognitive  strategy:  Provide  occasions  for  a  varietv  of  novel  problem 
solutions. 

•  Information:  Provide  verbal  links  to  additional  complexes  of 
information. 

•  Attitude:  Provide  additional  varied  situations  for  selected  choice  of 
action. 

•  Motor  skill:  Learner  continues  skill  practice. 

Merrill  (1997,  online  version)  describes  some  basic  instructional  strategies: 

•  Information-About  Strategy:  “describes  features,  cautions,  and  context  for  some  set 
of  entities,  activities,  or  processes”  (p.  2).  This  strategy  is  contrasted  with  and  used  to 
supplement  other  kinds  of  strategies. 

•  Parts  of ...  Strategy:  enables  the  learner  “to  identify  the  name  and  location  (with 
regard  to  some  whole)  of  a  given  part  of  a  device  or  system”  (p.  3). 

•  Concept  or  Kinds  of ...  Strategy:  enables  the  learner  “to  identify  unencountered 
examples  of  objects,  devices,  procedures,  actions,  or  symbols  as  belonging  to  a  particular 
class”  (p.  3). 

•  Procedure  Strategy:  enables  a  learner  to  perform  a  “series  of  actions  which  lead  to 
some  desired  consequence”  (p.  4). 

•  Process,  Principle  or  How  does  ...  work  Strategy:  helps  the  learner  to  predict  the 
consequence  of  an  event  given  a  set  of  conditions  or,  given  a  consequence,  to  be  able  to 
identify  the  conditions  which  lead  to  the  consequence. 

Compatible  with  the  general  framework  of  the  nine  instructional  events  (Gagne,  &  Briggs,  1979) 
and  Merrill’s  (1997)  instructional  strategies,  specific  approaches  have  been  recommended  to 
teach  different  types  of  content.  Summaries  of  these  approaches  are  provided  in  Appendix  A. 


60 


Technical  Report  2005-002 


The  Elaboration  Theory  of  Instruction.  Reigeluth  and  his  colleagues  have  developed  methods 
to  sequence  related  ideas  in  an  instructional  presentation.  The  Elaboration  Theory  of  Instruction 
(Reigeluth  &  Stein,  1983)  builds  on  the  work  of  Gagne  (1968),  Ausubel  (1968),  Bruner  (1960), 
Norman  (1973),  and  many  others.  It  prescribes  a  simple-to-complex  sequence  intended  to  help 
the  learner  understand  the  importance  of  the  different  ideas  presented,  their  context,  and 
relationships  between  them.  This  approach  helps  the  learner  make  the  instruction  meaningful 
and  therefore  easier  to  learn,  retain,  and  remember. 

The  central  concept  of  Elaboration  Theory  is  the  elaboration  sequence,  “in  which:  (1)  the 
general  ideas  epitomize  rather  than  summarize  the  ideas  that  follow;  and  (2)  the  epitomizing  is 
done  on  the  basis  of  a  single  type  of  content  (Reigeluth  &  Stein,  1983,  p.  343,  authors’ 
emphasis).  Epitomizing  is  not  a  preview  or  summary.  Rather,  it  presents  a  few  fundamental  and 
representative  ideas,  examples,  and  practice  opportunities  that  “convey  the  essence  of  the  entire 
content”  (p.  343).  Reigeluth  and  Stein  (1983)  provide  examples  of  epitomes  for  conceptual, 
theoretical,  and  procedural  content.  The  reader  should  consult  the  original  for  details  and 
guidance  on  the  application  of  Elaboration  Theory. 

The  ARCS  Model  of  Instruction.  Keller  (1983,  1987)  developed  a  motivational  model  of 
instruction.  It  is  based  upon  earlier  work,  such  as  field  theory  (Lewin,  1935),  social-learning 
theory  (Rotter,  1972),  and  expectancy- value  theory  (see  Steers  &  Porter  [1975]  for  a  review). 

The  model  focuses  on  how  to  gain  and  maintain  the  attention  of  the  learner  throughout  the 
instructional  process.  Keller’s  model  is  labeled  the  ARCS  model  referring  to  the  four  basic 
categories  of  motivational  conditions: 

•  Attention  (originally  labeled  interest )  “refers  to  whether  the  learner’s  curiosity  is 
aroused,  and  whether  this  arousal  is  sustained  over  time”  (Keller,  1983,  p.  395).  The 
model  refers  to  different  types  of  arousal  and  recommends  various  techniques  to  increase 
each  type.  Perceptual  arousal  can  be  increased  by  using  novel,  surprising,  incongruous 
and  uncertain  events.  Inquiry  arousal  can  be  increased  by  stimulating  information 
seeking  behavior  by  posing  questions  or  problems  or  having  the  learner  generate 
questions  or  problems  to  solve.  Interest  can  be  maintained  by  varying  the  elements  of 
instruction  (e.g.,  switch  between  information  presentation  and  learner  activities  that  allow 
them  to  interact  with  the  content). 

•  Relevance  “refers  to  the  learner’s  perception  of  personal  need  satisfaction  in  relation  to 
the  instruction,  or  whether  a  highly  desired  goal  is  perceived  to  be  related  to  the 
instructional  activity”  (p.  395).  Relevance  can  be  increased  by  providing  the  learner  with 
concrete  examples,  with  which  the  learner  is  familiar  (e.g.,  related  to  their  previous 
experiences  and  values).  Relevance  can  also  be  increased  by  presenting  goal  oriented 
statements  and  objectives  and  by  explaining  the  utility  of  the  instruction  for  the  learner’s 
present  and  future  uses. 

•  Confidence  (originally  labeled  expectancy)  “refers  to  the  perceived  likelihood  of 
success,  and  the  extent  to  which  success  is  under  learner  control”  (p.  395).  Confidence 
can  be  increased  by  designing  instruction  to  enable  the  learner  to  succeed.  The 
instruction  should  be  challenging,  but  not  overwhelming.  Learners  should  see  that  their 
expended  efforts  directly  influence  the  consequences  by  receiving  relevant  and 
meaningful  feedback  about  their  efforts.  Positive  learner  expectations  can  be  increased 
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by  helping  them  estimate  the  probability  of  success  by  presenting  performance 
requirements  and  evaluation  criteria. 

•  Satisfaction  “refers  to  the  combination  of  extrinsic  rewards  and  intrinsic  motivation, 
and  whether  these  are  compatible  with  the  learner’s  anticipations.  Satisfaction  can  be 
increased  by  providing  learners  with  opportunities  to  use  their  newly  acquired  knowledge 
or  skills  in  a  real  or  simulated  setting.  Then,  providing  feedback  with  either 
reinforcements  to  sustain  the  desired  behavior  or  remediation  to  correct  undesired 
behavior.  At  all  times,  learners  should  perceive  that  the  outcomes  of  their  efforts  are 
consistent  with  their  expectations. 

Keller  (1987)  further  subdivided  each  of  the  ARCS  categories  into  subcategories  and 
provided  suggestions  for  motivational  strategies  under  each  subcategory.  These  are  shown  in 
Appendix  B. 

Keller  (1987)  also  recommended  that  the  ARCS  model  be  applied  at  each  stage  of 
instructional  development.  He  offered  the  following  suggested  activities  during  each  of  the 
following  four  stages: 

•  Define: 

o  Classify  the  motivational  problem(s)  to  be  solved 
o  Conduct  an  audience  analysis  to  identify  motivational  gaps 
o  Prepare  motivational  objectives  that  identify  the  behavior,  conditions,  and  criteria 
that  apply 

•  Design: 

o  Generate  a  list  of  potential  motivational  strategies  for  each  of  the  motivational 
objectives 

o  Review  the  list  of  potential  strategies  and  select  the  ones  to  be  used.  The 
strategies  should  follow  five  guidelines:  (1)  they  should  not  take  up  too  much 
instructional  time,  (2)  they  should  not  detract  from  the  instructional  objectives, 

(3)  they  should  fall  within  the  time  and  money  constraints  of  the  development  and 
implementation  aspects  of  the  instruction,  (4)  they  should  be  acceptable  to  the 
audience,  and  (5)  they  should  be  compatible  with  the  delivery  system  (including 
the  instructor’s  personal  style  and  preferences). 

•  Develop: 

o  Prepare  any  special  materials  that  are  required 
o  Integrate  the  materials  with  the  instruction 

•  Evaluate: 

o  Conduct  development  try-out  (formative  evaluations) 
o  Assess  the  motivational  outcomes  (as  well  as  learning  outcomes).  Judge 

motivational  outcomes  using  direct  measures  (e.g.,  persistence,  intensity  of  effort, 
emotion,  and  attitude). 

Recent  research  has  tested  the  effects  of  designing  computer-assisted  instruction  (CAI)  based 
on  the  principles  of  the  ARCS  model  (Song  &  Keller,  2001).  Three  types  of  CAI  were  designed 
and  presented  to  different  groups  of  learners.  Motivationally  minimized  CAI  contained  the 
minimal  number  of  motivational  strategies.  Motivationally  saturated  CAI  included  a  large 
number  of  motivational  strategies  which  were  presented  to  all  learners,  regardless  of  their 
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motivational  needs.  Motivationally  adaptive  CAI  provided  motivational  strategies  targeted  to 
the  immediate  needs  of  each  learner.  In  this  condition,  motivated  learners  were  not  presented 
with  motivational  strategies  that  might  distract  them.  The  learners’  levels  of  motivation  and 
attention  were  continuously  assessed  during  instruction.  Results  showed  that  the  motivationally 
adaptive  CAI  was  more  effective  than  either  of  the  other  approaches. 

Structure  of  Content.  The  purpose  of  instruction  is  to  provide  new  knowledge  to  the  learner. 
This  knowledge  must  be  structured  in  a  manner  that  can  be  used  at  a  later  time.  Citing  several 
researchers,  Glaser  and  Bassok  (1989)  state  that  “the  way  knowledge  is  structured  influences  its 
accessibility,  and  knowledge  representation  determines  understanding  and  influences  problem 
solving  (p.  633).  Thus,  the  way  the  learner  structures  his  or  her  knowledge  is  affected  by  the 
structure  of  the  instructional  content.  The  information  in  the  instructional  content  should  be 
presented  in  a  consistent  style  in  terms  of  writing  (e.g.,  voice,  grammar,  and  notations),  use  of 
instructional  media,  and  appearance  (see  the  User  Interface  Section).  In  addition,  there  are 
several  design  techniques  that  can  improve  the  structure  of  instructional  content. 

Boettcher  (2003)  defines  structure  as  “something  arranged  in  a  definite  pattern  of 
organization”  (p.l).  She  maintains  that  well  structured  content  supports  the  building  of 
foundations  for  complex  knowledge,  helping  the  learner  to  integrate  isolated  facts  into  more 
complex,  organized  knowledge.  This  is  an  important  point  that  is  sometimes  lost  in  the  rush  to 
include  all  content.  It  should  be  organized  to  allow  the  learner  to  gradually  progress  from  the 
basic  foundations  of  information  to  the  organization  of  more  and  more  complex  material  (e.g.,  as 
recommended  in  Elaboration  Theory). 

Merrill  (1983)  believes  that  “associative  memory  is  composed  of  a  hierarchical  network 
structure”  (p.  302).  Any  basic  concept  that  is  learned  may  be  related  to  other  concepts  that  are 
either  more  specific  (subordinate)  or  more  general  (super-ordinate).  Associative  memory  uses 
the  relationships  among  these  categories  to  both  store  and  retrieve  information.  For  example,  the 
basic  concept  “carrot”  is  a  subordinate  concept  of  the  category  “vegetables”  and  a  super-ordinate 
concept  of  the  category  “baby  carrots.”  Learning  to  distinguish  between  and  use  the 
relationships  among  these  categories  helps  in  both  learning  and  memory. 

Nitsch  (1977)  provides  support  for  the  idea  that  learning  is  facilitated  by  structuring  it  to  use 
the  hierarchical  nature  of  memory.  Like  Bransford  and  his  colleagues  (1986;  1989),  she  believes 
that  the  initial  instruction  on  a  topic  should  be  structured  so  that  learners  can  establish  a  firm 
contextual  basis  for  understanding  the  new  concept  (see  Criterion  l.f).  Nitsch  (1977)  found  that 
providing  concrete  examples  in  additional  to  definitions  “can  have  a  positive  influence  upon 
subjects’  subsequent  ability  to  use  their  knowledge  of  the  concepts”  (p.  15).  She  refers  to  this  as 
the  “contextualization”  of  knowledge.  She  also  found  that  providing  a  variety  of  examples  in 
different  contextual  domains  “resulted  in  a  much  more  flexible  ability  to  use  the  conceptual 
information”  (p.  37).  This  contextual  variety  made  initial  learning  more  difficult  but  facilitated 
the  learners’  ability  to  “decontextualize”  the  knowledge  so  they  could  apply  it  in  new  situations 
and  different  contexts.  Thus,  content  should  be  structured  to  help  the  learner  create  the 
conceptual  hierarchy  that  will  assist  in  retrieving  and  using  learned  concepts. 
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Structural  Areas  of  Content.  Boettcher  (2003)  discusses  four  structural  areas  of  content  and 
observes  that  there  are  nuances  of  interaction  between  all  four. 

1.  The  first  structural  area  involves  presentation  of  “content  for  which  an  organization  is 
clearly  visible  and  in  which  concepts  are  presented  clearly  and  precisely”  (p.l). 

2.  The  presentation  should  be  semantically  well  structured  to  support  instruction  (e.g., 
simple  concepts  are  fully  explained  and  then  used  as  building  blocks  to  explain  more 
complex  concepts). 

3.  The  content  should  be  structured  to  support  learner  needs  (e.g.,  presented  at  the  reading 
level  of  the  audience). 

4.  Finally,  it  should  be  technologically  well  structured  (e.g.,  technology  is  not  used 
gratuitously,  but  only  to  support  content). 

Below  are  brief  descriptions  of  some  techniques  that  can  be  used  to  structure  content  to  help 
learners  learn  more  effectively. 

Advance  Organizers.  One  of  the  techniques  that  help  learners  easily  see  the  organization  of 
the  content  is  to  use  advance  organizers.  Advance  organizers  are  “subsuming  concepts” 
(Ausubel,  1960,  p.  267)  that  help  learners  mobilize  most  relevant  aspects  of  the  instructional 
information  to  create  the  cognitive  structures  that  facilitate  learning  and  memory.  They  are 
“concepts  introduced  before  material  is  covered  which  are  on  a  more  general  or  abstract  level 
than  the  material  which  is  to  be  covered”  (Chesebro  &  McCroskey,  1998,  p.  449).  Ausubel 
(1960,  pp.  269-271)  demonstrated  that  advance  organizers,  such  as  background  information  that 
is  more  concrete  than  the  abstract  information  that  may  follow  it,  improved  learners’  memory. 

Advance  organizers  provide  a  conceptual  structure  or  framework  to  aid  the  encoding  of 
information  and  also  to  help  the  learner  activate  prior  knowledge  and  relate  it  to  the  new 
information.  They  are  especially  useful  in  helping  the  learner  make  distinctions  and 
comparisons  between  similar  concepts. 

Ausubel  (1968)  identified  two  types  of  advance  organizers: 

1.  Expository  Organizers:  Are  used  with  unfamiliar  material  to  set  the  stage  for  learning 
new  concepts.  For  example,  when  teaching  the  American  Revolution,  introduce  stages 
common  to  all  revolutions  as  a  framework  for  the  new  information. 

2.  Comparative  Organizers:  Are  used  to  aid  the  integration  of  new  ideas  by  comparing  and 
contrasting  them  with  similar  concepts  that  have  already  been  learned.  For  example, 
atomic  structure  can  be  taught  by  discussing  the  similarities  and  differences  between 
electrons  circling  a  nucleus  and  moons  orbiting  around  a  planet. 

Ausubel  and  Youssef  (1963)  compared  the  effects  of  comparative  organizers  using  two 
groups  of  learners.  They  gave  one  group  a  comparative  organizer  (a  brief  textual  comparison  of 
the  relationships  between  Christianity  and  Buddhism)  prior  to  assigning  both  groups  a  long 
reading  assignment  on  Buddhism.  Later,  the  group  that  received  the  comparative  organizer 
recalled  more  of  the  text. 

Use  of  Signals  and  Cues.  “A  signal  is  any  technique  used  to  make  some  elements  of  a  text  or 
document  more  salient  than  other  elements”  (Clark,  R.  C.,  2003,  p.  74).  Signals  can  include 
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topic  headings,  typographical  cues  like  italics  or  color,  overviews,  outlines,  highlighting,  and 
pointer  phrases  such  as  results  in...,  therefore...,  or  because  of...  Lorch  and  Lorch  (1996) 
compared  the  learning  effects  of  signaled  and  unsignaled  text.  They  found  that  the  use  of  signals 
helped  learners  recall  more  from  complex  text,  but  not  from  simple  text.  This  finding  has  also 
been  extended  to  the  use  of  signals  in  narration  (Mautone  &  Mayer,  2001). 

Use  of  Summaries.  Another  technique  is  to  use  summaries  at  the  end  of  each  section  of 
material  and  reviews  throughout  the  course.  These  help  the  learner  to  perceive  the  patterns 
among  different  pieces  of  information  presented  in  the  course. 

Activating  Existing  Knowledge.  It  is  widely  accepted  that  learning  requires  that  new 
knowledge  be  presented  in  a  way  that  activates  and  builds  on  existing  knowledge  (Bower,  Clark, 
Lesgold,  and  Winzenz,  1969;  Halpem  and  Hakel,  2003).  This  is  true  for  both  the  knowledge  that 
a  learner  brings  to  the  course  and  also  for  the  new  knowledge  that  is  gained  during  the  course.  It 
can  be  assumed  that  learners  taking  an  introductory  course  will  know  very  little  about  the  subject 
matter.  However,  they  still  may  bring  misconceptions  about  topics  in  the  course  and  these 
misconceptions  may  have  to  be  corrected  prior  to  presenting  new  information.  Learners  taking  a 
refresher  course  will  know  much  more  about  the  subject  matter,  but  may  need  to  reorganize  that 
knowledge  for  new  purposes  or  update  it  based  upon  new  insights  about  the  topics.  In  either 
case,  the  information  presented  in  the  course  should  be  structured  to  help  the  learner  expand 
existing  information  structures  and  build  new  ones. 

Helping  the  learner  build  on  existing  knowledge  can  help  to  avoid  the  distractions  of 
interesting,  but  non-relevant  instructional  information.  Gamer,  Gillingham,  and  White  (1989) 
tested  the  effect  of  “seductive  details”  (interesting  by  non-relevant  information)  on  the  ability  of 
adults  and  seventh  graders  to  remember  important  details  from  a  paragraph  of  text.  They  found 
that  the  seductive  details  reduced  the  performance  of  both  groups.  However  the  adults  were  still 
able  to  remember  some  important  details.  These  researchers  assumed  that  the  difference 
between  adults  and  children  is  because  the  adults  had  more  extensive  domain  knowledge.  This 
previous  domain  knowledge  enabled  them  to  more  readily  classify  the  new  information  as 
important  or  simply  interesting.  Instructional  materials  should  be  designed  to  minimize 
“seductive  details”  while  providing  a  logical  structure  that  relates  the  new  material  to  the 
learners’  existing  knowledge. 

An  instructional  technique  that  has  been  shown  to  help  activate  a  learner’s  existing 
knowledge  base  is  the  use  of  analogies  (Stepich  &  Newby,  1988).  This  technique  is  discussed  in 
the  context  of  instructional  activities  (Criterion  2. a). 

Levels  of  Content  Structure.  Similar  to  Rosch’s  (1978)  categorization  approach,  Boettcher 
(2003)  believes  that  job-relevant  knowledge  and  skills  may  require  one  of  three  different  levels 
of  content  structure  (also  see  Criterion  l.f). 

•  The  first  level  presents  core  concepts  and  principles. 

•  The  second  level  includes  content  with  well-structured  problems  with  known  solutions. 

•  The  third  level  consists  of  content  with  less-structured,  complex  problems  without 
known  solutions. 
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Ideally,  as  learners  advance  from  novice  to  expert,  they  will  proceed  through  these  three  levels, 
gradually  building  their  understanding  by  engaging  in  learning  activities  at  each  level.  Table  4 
lists  some  of  the  structural  characteristics  and  requirements  at  each  level  of  content  (Boettcher, 
2003). 


Table  4: 

Structural  Characteristics  of  Three  Levels  of  Content 


Content  Level 

Structural  Characteristics 

Level  one:  Core  concepts  and 
principles 

•  Provides  descriptions  of  core  concepts  (may  use  various 
media  dynamically) 

•  Provides  means  for  learners  to  rehearse  and  interact  with 
processes 

•  Helps  learner  cluster  and  chunk  information 

•  Clearly  shows  relationships  and  patterns  among  core  concepts 

•  Reveals  differentiations  and  distinctions  among  concepts 

•  Links  core  concepts  with  current  issues  in  job  area 

Level  two:  Well-structured 
problems  with  known  solutions 

•  Presents  consistent  elements  of  the  problem  sets 

•  Reveals  patterns  inherent  in  the  problems 

•  Reveals  sources  and  types  of  knowledge  that  contribute  to 
problem  solutions 

•  Chunks  and  clusters  the  elements  of  the  solutions 

•  Gradually  increases  the  complexity  of  the  applied  rules  and 
principles 

•  Engages  the  learner  in  the  solutions  (perhaps  through  the  use 
of  simulations  or  other  interactive  activities) 

Level  three:  Less-structured, 
complex  problems  without  known 
solutions 

•  Provides  complex,  real-world  scenarios  and  case  studies 

•  Engages  the  learners  in  solving  complex  problems  where 
neither  the  elements  or  solutions  are  known  (e.g.,  through  use  of 
simulations  nor  other  activities  that  involve  complex 
interactions  among  problem  elements  or  interrelated  problems) 

•  Provides  explanations  of  real-life  problems  from  experts, 
including  testimonials  on  their  “solutions” 

Scaffolding.  A  teaching  strategy  that  addresses  how  to  help  learners  build  on  prior 
knowledge  and  internalize  new  information  is  called  scaffolding.  It  is  based  on  Lev  Vygotsky’s 
sociocultural  theory  (Van  Der  Stuyf,  2002).  Vygotsky  advocated  that  learners  be  supported  at 
their  current  level  of  competence  while  encountering  new  information.  These  temporary 
supports  or  scaffolds  are  withdrawn  as  the  learner’s  abilities  increase  until  he  or  she  is  able  to 
complete  the  task  or  master  the  concepts  independently. 

McKenzie  (1999)  describes  eight  characteristics  of  scaffolding. 

1 .  Provides  clear  directions  to  reduce  learners’  confusion  about  what  they  must  do  to  meet 
the  expectations  of  the  learning  activity. 
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2.  Clarifies  purpose  by  helping  learners  understand  why  the  problem,  issue  or  decision  is 
important  and  why  they  should  care  about  it. 

3.  Keeps  learners  on  task  by  providing  a  pathway  through  the  progression  of  activities. 

4.  Clarifies  expectations  by  providing  examples  of  quality  work,  rubrics,  and  standards  that 
define  quality  work. 

5.  Directs  learners  to  worthy  sources  so  they  can  avoid  confusing,  weak,  and  unreliable 
information. 

6.  Reduces  uncertainty,  surprise,  and  disappointment  by  testing  lessons  to  eliminate 
distractions  and  frustrations  to  the  extent  possible. 

7.  Provides  efficiency  by  helping  focus  the  learner  on  the  work  effort. 

8.  Creates  momentum  by  concentrating  and  directing  the  learner’s  energy  toward 
accumulating  insight  and  understanding. 

Metaphors.  Remelhart  and  Norman  (1981)  maintain  that  analogical  reasoning  is  the  most 
common  way  that  people  apply  knowledge  learned  in  one  domain  to  another  domain  (e.g.,  one 
task  to  another).  One  technique  that  can  assist  the  learner  apply  such  analogical  reasoning  is  to 
use  an  intuitive  metaphor  for  the  learner  to  interact  with  the  content  (Reeves,  1994  as 
summarized  in  Botha,  2004).  These  metaphors  might  be  books,  bookshelves,  space  exploration, 
buildings  with  different  rooms,  college  campuses,  or  many  others.  Jones  and  Okey  (1995) 
recommend  that  the  metaphor  should  reflect  the  nature  of  the  content.  For  example,  a  student 
learning  auto  repair  could  access  various  lessons  by  clicking  on  a  portion  of  an  automobile  (e.g., 
the  fuel  system,  the  transmission,  the  brakes).  Henke  (2001)  recommends  that  the  metaphor  be 
designed  to  be  obvious  to  the  learner.  He  also  recommends  that  the  course  should  use  navigation 
styles  that  are  consistent  with  the  metaphor.  Chapter  7  (Evaluation  Area  5,  Navigation  & 
Operation)  provides  discussions  of  the  scales  that  evaluate  the  design  of  the  user  interface  to 
better  support  learning. 

Learner  Control.  The  ability  for  learners  to  control  the  presentation  of  content  is  often 
described  as  one  of  the  potential  advantages  of  computer-based  and  web-delivered  instruction 
(e.g.,  Hannafin  and  Sullivan,  1995).  However,  evidence  from  a  wide  variety  of  studies 
(Niemiec,  Sikorski,  and  Walberg,  1996)  indicates  that  “as  the  extent  or  learner  control  over 
various  aspects  of  instruction  increases,  learning  may  decrease”  (Clark,  R.,  2003,  p.  14).  Some 
studies  do  show  that  there  are  benefits  of  some  limited  forms  of  learner  control,  such  as  control 
over  the  pacing  of  a  multimedia  presentation  (Doherty,  1998;  Mayer  &  Chandler,  2001; 
additional  information  is  provided  in  the  discussion  of  Criterion  6.e,  use  of  multimedia).  Course 
designers  need  to  provide  learner  control  of  pacing  and  other  types  of  “surface”  control,  but 
retain  the  control  of  the  order  of  instructional  topics  and  instructional  activities. 

Size  of  Instructional  Segments.  Many  recommend  that  the  instructional  content  should  be 
organized  and  presented  in  small,  manageable  steps  (Norman,  1976;  Dick  and  Carrey,  1996; 
Texas  Education  Agency,  2003;  Trollinger,  2003). 

Jones  and  Okey  (1995)  explain  that  instructional  segments  should  be  small  so  the  learner  is 
not  overwhelmed  by  too  much  information.  The  learners  need  to  be  able  to  gradually  integrate 
the  new  information  into  their  existing  knowledge  and  build  new  knowledge  structures.  Jones 
(1989)  refers  to  the  presentation  of  content  in  small  segments  “progressive  disclosure.” 
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Therefore,  only  the  material  that  is  relevant  to  the  specific  objective  of  the  segment  should  be 
presented  at  any  one  time.  The  learner  should  know  when  a  segment  has  been  chosen  and  when 
it  has  been  completed  (l.i).  Furthermore,  it  should  be  readily  apparent  to  learners  if  they  have 
completed  all  of  the  prerequisite  topics  prior  to  beginning  a  new  segment  of  instruction  (see  1  .h). 
This  approach  reduces  the  perceived  complexity  of  the  content  and  allows  the  learner  to  reflect 
on  its  meaning. 

Sequencing  Learning  Objectives.  Mayer,  Mathias  and  Wetzell  (2002)  demonstrated  the  value 
of  teaching  related  concepts  prior  to  the  task  or  process  that  is  the  focus  of  the  lesson.  They 
preceded  a  multimedia  lesson  on  the  mechanical  processes  of  how  brakes  work  with  pre-lessons 
on  each  individual  component  in  the  braking  system.  No  matter  what  medium  was  used  (e.g., 
text  or  multimedia),  learning  was  better  with  the  pre-lesson  than  when  all  of  the  information  was 
presented  at  once. 

The  handbook  on  Instructional  Systems  Development  (Department  of  Defense,  2001) 
provides  guidance  on  how  to  structure  a  course  in  terms  of  the  sequence  and  hierarchy  of 
learning  objectives  (LOs).  The  following  discussion  is  a  very  brief  summary  of  portions  of  this 
guidance.  Table  5  shows  some  of  the  possible  relationships  among  learning  objectives. 


Table  5: 

Types  of  Relationships  Among  Learning  Objectives 

(Modified  from:  Department  of  Defense,  2001,  pp.  86-88) 


Dependent 

Conflicting 

Supportive 

Independent 

Knowledge  and  skills  in 
one  LO  are  prerequisite 
to  those  in  the  other  LO. 

Knowledge  and  skills  in 
one  LO  conflict  in  some 
respect  with  those  in 
another  LO. 

Knowledge  and  skills 
learned  in  one  LO  makes 
learning  easier  in  the  other 
LO. 

Knowledge  and  skills  in  one  LO 
are  unrelated  to  those  in  the  other 
LO. 

To  master  one  of  the 

LOs,  it  is  first  necessary 
to  master  the  other. 

Mastering  one  LO  may 
cause  difficulty  in 
mastering  the  other  LO. 

Mastering  one  LO  transfers 
to  other,  making  learning 
of  the  other  easier. 

Mastering  one  LO  does  not 
simplify  mastering  the  other. 

LOs  must  be  arranged  in 
the  sequence  indicated 
by  the  knowledge  and 
skills  hierarchy. 

LOs  must  be  taught  closely 
together,  directly 
addressing  the  conflicting 
elements  between  the  two 
LOs. 

LOs  should  be  placed  close 
together  in  the  sequence  to 
permit  optimum  transfer  of 
learning  from  one  LO  to 
the  other. 

In  general,  the  LOs  can  be 
arranged  in  any  sequence  without 
loss  of  learning  efficiency. 

When  evaluating  a  completed  course  or  plans  for  course  development  (e.g.,  storyboards),  the 
evaluator  should  consider  the  following  issues  (additional  details  can  be  found  in  the  handbook). 
Since  most  tasks  are  made  up  of  subtasks,  it  is  important  that  learners  learn  the  subtasks  in  the 
proper  sequence.  A  learning  hierarchy  is  a  “map  that  depicts  the  relationship  of  various  LOs  and 
their  sequence”  (p.  83).  This  hierarchy  specifies  the  most  effective  and  efficient  learning 
sequence. 
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Early  in  content  development,  each  learning  object  (for  all  tasks  and  subtasks)  needs  to  be 
classified  as  either  a  Terminal  Learning  Objective  (TLO;  what  the  learner  will  accomplish  when 
successfully  completing  the  instruction  for  a  task)  or  an  Enabling  Learning  Objective  (ELO; 
what  the  learner  must  attain  in  order  to  accomplish  a  terminal  objective). 

Next,  the  ELOs  and  TLOs  are  clustered  or  grouped  into  “logical  and  meaningful  portions  of 
training  such  as  units,  lessons,  or  segments”  (p.  85).  Clustering  of  LOs  can  be  based  upon  those 
requiring  common  prerequisites,  those  with  relationships  to  the  same  system  or  type  of  action; 
those  requiring  common  knowledge  or  skills;  or  those  requiring  the  same  learning  approach 
(e.g.,  taught  by  demonstration). 

Then,  the  LOs  are  sequenced  within  the  course  using  some  combination  of  the  following 
techniques:  job  performance  order,  chronological  order,  cause  and  effect  order;  critical  order, 
simple-to-complex  order,  complex-to-simple  order,  or  known-to-unknown  order. 

Relationships  among  LOs  can  also  be  used  to  logically  sequence  a  course.  “The  four  most 
important  relationships  in  sequencing  are  1)  dependent,  2)  supportive,  3)  independent,  and  4) 
conflicting”  (p.  86).  Table  5  provides  a  summary  of  these  relationships. 

For  dependent  LOs,  it  is  important  that  the  content  be  structured  to  present  the  material  in  the 
appropriate  sequence  (see  Criterion  1  .h).  On  the  other  hand,  supportive  LOs  should  be 
integrated  with  each  other  in  instructional  events  or  activities.  As  Malcolm  (2000)  observes, 
instructional  events  are  often  “not  integrated  with  each  other  even  when  they  support  the  same 
performance  need”  (p.  3).  He  offers  an  example  of  instruction  for  customer  service 
representatives.  These  courses  often  teach  telephone  skills  and  application  system  skills 
separately,  even  though  the  real  job  requires  them  to  be  performed  simultaneously.  This  is,  as  he 
paraphrases  Gloria  Geiy,  “like  giving  people  the  threads  of  learning  and  expecting  them  to  knit 
together  competent  on-the-job  performance”  (p.3).  Content  should  be  designed  to  do  this 
weaving  for  learners  so  they  can  experience  learning  and  doing  the  interwoven  tasks  that  are 
more  representative  of  the  real  work  they  will  have  to  do. 

Relationships  Among  Models  of  Instruction  and  Evaluation  Criteria.  Educational  researchers 
have  proposed  various  models  of  how  to  organize  instructional  sequences.  As  discussed  above, 
different  models  focus  on  different  aspects  of  the  instructional  process.  Regardless  of  the  model 
they  choose,  most  instructional  developers  agree  that  teaching  different  types  of  knowledge  and 
skills  requires  different  conditions  for  learning  (Gagne,  1985).  “If  an  instructional  experience  or 
environment  does  not  include  the  instructional  strategies  required  for  the  acquisition  of  the 
desired  knowledge  or  skill,  the  effective,  efficient,  and  appealing  learning  of  the  desired  outcome 
will  not  occur  (Merrill,  2000,  p.  1). 

Table  6  shows  how  the  instructional  models  discussed  above  relate  to  the  nine  events  of 
instruction  and  to  some  of  the  instructional  evaluation  criteria.  Although  the  nine  events  appear 
to  indicate  a  sequential  approach  to  instruction,  in  fact,  instruction  is  much  more  of  an  iterative 
process.  For  example,  gaining  attention  is  necessary  at  the  beginning  of  instruction,  but  this 
attention  must  be  maintained  throughout  the  entire  instructional  process.  Likewise,  feedback 
should  not  be  presented  solely  at  the  end  of  the  instruction,  but  rather  as  an  ongoing  process 
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throughout.  Appendix  A  summarizes  the  4  types  of  content  in  Merrill’s  Component  Response 
Theory.  The  reader  should  consult  Merrill  (1983)  for  a  discussion  of  his  4  presentation  forms 
and  3  types  of  performance.  Elaboration  Theory  and  the  ARCS  Model  are  discussed  above. 
Additional  details  on  the  ARCS  Model  can  be  found  in  Appendix  B. 


Table  6: 

Instructional  Models  and  Evaluation  Criteria 


Nine  Events  of 
Instruction 

(Gagne  &  Briggs, 
1979) 

Instructional 
Evaluation  Criteria 

ARCS 

Model 

(Keller,  1983, 
1987) 

Component 

Response 

Theory 

(Merrill,  1983) 

Elaboration 

Theory 

(Reigeluth  & 
Stein,  1983) 

1.  Gaining  Attention 

1  .b.  The  purpose  of  the 
course  is  clearly  stated. 

HRiSSiluH 

2.  Informing  learner 
of  objective 

1  .c.  The  instructional 
objectives  are  clearly 
stated. 

Relevance 

3.  Stimulating  recall 
of  prerequisites 

1  .h.  There  are  clear 
indications  of 
prerequisites. 

4.  Presenting  the 
stimulus  material 

1  .a.  The  content  is 
presented  in  a  logical 
manner. 

4  types  of  content 
(facts,  concepts, 
procedures,  & 
principles;  see 
Appendix  A) 

Epitomizing: 
layers  of 
instruction 

5.  Providing 
learning  guidance 

2.a.  Activities  are  relevant 
2.c.  Varied  instructional 
methods 

4  presentation 
forms  (general 
expository,  general 
inquisitory, 
instance 
expository,  & 
instance 
inquisitory) 

6.  Eliciting  the 
performance 

2.a  Activities  are  relevant 
2.b.  Learner  is  required  to 
interact  with  content 

Confidence 

(Expectancy) 

3  types  of 
performance  (find, 
use,  &  remember) 

7.  Providing 
feedback 

4.  Performance  feedback 
(all  criteria  in  section  4) 

8.  Assessing 
performance 

3.  Performance 
assessment  (all  criteria  in 
section  3) 

9.  Enhancing 
retention  and 
transfer 

2.a.  Activities  are  relevant 
4.b.  Feedback  is 
meaningful 

Satisfaction 

For  the  development  of  instructional  products,  it  is  also  important  that  logical  decisions  be 
made  about  other  instructional  issues.  For  example,  specific  instructional  activities  or 
instructional  media  should  be  chosen  logically.  These  issues  are  covered  under  other  evaluation 
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criteria  (e.g.,  Evaluation  Area  2  on  instructional  activities  [Chapter  3]  and  Evaluation  Area  6  on 
content  presentation  [Chapter  8]). 


l.b.  The  Purpose  of  the  Course  is  Clearly  Stated 

The  purpose  of  a  course  sets  the  stage  for  both  learners  and  developers.  A  learner  should  not 
have  to  guess  why  he  or  she  is  taking  a  course  or  how  the  course  will  support  his  or  her  job 
performance. 

Using  the  Scale 

This  scale  rates  the  course  in  terms  of  how  specifically  the  purpose  of  the  course  is 
communicated  to  the  learner.  Evaluators  should  use  the  anchor  statements  and  example  purpose 
statements  to  help  assign  a  rating  on  this  criterion. 

The  scale  can  also  be  used  during  course  development.  It  can  sensitize  the  developer  to  the 
importance  of  purpose  statements  and  help  them  make  the  statement  specific,  relevant,  and  clear. 
Additional  information  to  help  the  evaluator  understand  this  criterion  is  provided  in  the 
discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

Understanding  the  purpose  or  a  course  enables  learners  to  establish  the  relevance  and 
appropriateness  of  the  course  (Bostock,  2004,  citing  Tweddle,  Avis  and  Waller,  1998).  This 
information  should  be  presented  at  the  beginning  of  the  course  as  a  general,  “high  level” 
declaration  of  who  should  take  the  course  (e.g.,  novices,  journeymen,  or  experts;  persons  from  a 
particular  job  series;  persons  wishing  to  change  jobs),  what  is  to  be  accomplished  in  the  course 
(e.g.,  what  general  topics  will  be  covered),  why  they  should  take  the  course  (e.g.,  how  it  will 
improve  their  job  performance),  and  the  benefits  they  will  gain  from  the  course  (e.g.,  help 
advance  their  career  or  help  them  obtain  certification).  The  “high-level”  messages  in  the 
purpose  statement  should  be  used  to  guide  the  development  of  more  specific  learning  objectives 
(see  l.c). 

The  purpose  of  a  course  also  has  implications  for  how  content  is  designed  and  presented.  For 
example,  if  the  purpose  is  to  provide  initial  instruction,  the  learner  will  need  the  content  to  be 
presented  in  a  very  structured  manner  that  ensures  that  each  new  piece  of  information  is 
integrated  with  previous  information  and  mental  structures.  On  the  other  hand,  refresher 
instruction,  where  the  learner  only  needs  a  specific  piece  of  information  to  accomplish  a  specific 
task  or  has  forgotten  something  needed  on  his  or  her  job,  might  be  designed  to  allow  a  learner  to 
browse  to  find  that  information. 

There  is  a  considerable  amount  of  empirical  data  that  indicate  the  importance  of  a  clearly 
stated  purpose  for  the  course.  The  first  major  step  in  learning  is  the  forming  of  meaningful 
associations  among  pieces  of  information  (Miller,  1978).  The  context  in  which  information  is 
presented  has  a  major  effect  on  how  these  associations  are  formed  (Miller,  Heise,  &  Lichten, 
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1951;  Klatsky,  1975).  These  associations  must  be  deeply  processed  (encoded  into  long-term 
memory;  LTM),  which  is  believed  to  consist  of  a  network  of  associations  (Brown  &  McNeill, 
1966;  Craik  &  Lockhart,  1972;  Anderson  &  Bower;  1972;  1973;  1974).  Furthermore,  it  is 
widely  recognized  that  humans  can  only  attend  to  a  limited  amount  of  information  at  any  one 
time  (Broadbent,  1963;  Klatsky,  1975).  Therefore,  humans  must  select  the  information  on  which 
they  focus  their  attention  (Norman,  1976).  It  is  therefore  critical  that,  during  the  instructional 
process,  the  learner’s  attention  is  directed  toward  the  critical  information  that  must  be  learned.  A 
clearly  stated  purpose  for  the  course  is  the  first  step  in  this  direction  of  attention.  It  is  so 
important  that  Gagne  and  Briggs  (1979)  identify  it  as  one  of  their  9  events  of  instruction. 

A  clearly  stated  purpose  can  serve  as  an  “advance  organizer”  (Ausubel,  1960;  Mautone  & 
Mayer,  2001).  It  prepares  the  learner  to  attend  to  the  important  details  that  will  follow.  A 
clearly  stated  purpose  statement  can  also  help  learners  filter  out  “seductive  details”  that  can 
interfere  with  the  learner’s  attention  to  important  instructional  information  (Gamer,  et  al.,  1989; 
Harp  &  Mayer,  1998).  The  effects  of  “seductive  details”  were  also  discussed  under  Criterion  l.a 
and  will  be  discussed  under  Criterion  6.e  on  the  use  of  multimedia.  The  discussion  under 
Criterion  6.e  highlights  the  need  to  avoid  irrelevant,  extraneous  words,  pictures,  and  sounds.  It 
is  easier  to  identify  such  irrelevant  information  if  the  purpose  of  the  course  is  clearly  stated. 

A  clearly  stated  purpose  is  especially  important  for  adult  learners.  Some  educators  believe 
that  adults  are  better  able  to  engage  in  self-directed  learning  than  younger  learners  (e.g., 

Knowles,  1989;  Hiemstra  &  Sisco,  1990;  Brookfield,  1995).  One  of  the  assumptions  behind  this 
belief  is  that  adult  learners  are  “problem-centered  and  interested  in  immediate  application  of 
knowledge”  (Merriam,  2001,  p.  5).  Thus,  a  clearly  stated  purpose  will  help  the  adult  learner 
focus  and  maintain  his  or  her  interest  and  attention  on  the  course. 

Another  area  which  supports  the  need  for  a  clearly  stated  purpose  is  research  on  motivation. 
Learners  must  be  motivated  to  learn  and  one  of  the  strongest  motivators  is  an  objective  and 
obtainable  goal  (Locke  &  Bryan,  1966;  Magill,  1980).  The  relevance  of  the  material  is  also  a 
factor  in  motivation  (Ausubel,  1968).  Thus,  a  well  stated  purpose  for  the  course  helps  the 
learner  begin  the  process  of  setting  goals  and  establishing  the  relevance  of  the  material  to  be 
learned. 


l.c.  The  Instructional  Objectives  are  Clearly  Stated 

An  instmctional  objective  (often  called  a  learning  objective  or  LO)  is  “a  statement  of  what  the 
learner  will  do  after  the  lesson  or  module  to  demonstrate  that  they  [sic]  have  understood  the 
content  or  acquired  the  skills  included  in  the  lesson”  (Clark,  R.  C.,  2003,  pp.  73-74).  Another 
definition  of  an  LO  focuses  on  the  performance  requirement  after  the  learner  has  completed  the 
objective,  “A  LO  is  a  precise  statement  of  the  capability...  a  student  is  expected  to  demonstrate, 
the  condition  under  which  the  KSA  is  to  be  exhibited,  and  the  standard  of  acceptable 
performance”  (Department  of  Defense,  2001,  p.  78). 
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Using  the  Scale 

This  scale  is  used  to  determine  how  well  the  instructional  objectives  are  clearly  stated.  The 
anchor  statements  below  the  scale  should  be  used  to  help  determine  the  score  to  assign  on  this 
criterion.  Additional  guidance  is  provided  in  the  discussion  below.  Remember,  if  you  assign  a 
rating  of  one  (1)  on  this  criterion,  you  must  also  assign  a  rating  of  one  (1)  on  Criterion  l.d. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

The  instructional  objectives  should  follow  from  and  support  the  “high-level”  purpose  of  the 
course  (see  l.b.).  They  should  provide  more  specificity  in  terms  of  how  the  learner  will  benefit 
from  each  course  module,  unit,  lesson,  or  topic.  Nash  (2003)  adds  another  requirement:  that  they 
describe  how  the  learner  will  achieve  the  objective.  Finally,  the  objectives  should  also  help 
learners  relate  different  course  topics  to  one  another,  so  they  understand  how  their  progression 
through  the  course  helps  them  build  and  expand  on  their  knowledge  and  skills. 

Gagne  and  Briggs  (1979)  argue  that  a  clearly  stated  instructional  objective  is  an  important 
step  in  gaining  and  maintaining  the  learner’s  attention.  It  is  so  important  that  they  call  it  out  as 
the  second  of  their  nine  events  of  instruction  (see  Table  3). 

As  discussed  above  (l.b.),  adult  learners  are  especially  interested  in  the  objectives  of  a  course 
or  a  course  segment.  These  objectives  can  be  made  more  relevant  if  they  indicate  how  they  will 
benefit  the  adult  learners  in  their  jobs  or  careers.  As  such,  they  can  also  serve  as  “advance 
organizers”  (Ausubel,  1960)  to  help  learners  structure  the  information  for  better  encoding  and 
retrieval. 

Much  of  the  research  supporting  the  need  for  clearly  stated  instructional  objectives  is  the 
same  as  those  studies  that  supported  the  need  for  a  clearly  stated  purpose  for  the  course  (l.b). 

This  research  includes  the  need  for  information  to  be  relevant  (Ausubel,  1968),  motivating 
(Locke  &  Bryan,  1966;  Magill,  1980),  and  capable  of  holding  the  learner’s  attention  (Norman, 
1976).  Additional  empirical  support  comes  from  data  on  the  formation  of  associations  (Miller,  J. 
G.,  1978)  and  the  context  in  which  information  is  presented  (Miller,  G.  A.,  Heise,  &  Lichten, 
1951;  Klatsky,  1975).  A  clearly  stated  instructional  objective  can  aid  the  learner  to  deeply 
process  the  associations  and  encode  them  into  long-term  memory  (LTM).  As  several  studies 
have  shown  (Brown  &  McNeill,  1966;  Craik  &  Lockhart,  1972;  Anderson  &  Bower;  1972; 

1973;  1974)  more  deeply  processed  information  is  more  readily  remembered.  In  addition, 
clearly  stated  instructional  objectives  also  aid  learners  to  attend  to  the  most  important 
information  that  must  be  learned  (Broadbent,  1963;  Klatsky,  1975;  Norman,  1976). 

Goal-setting  theory  (Locke,  2000)  argues  that  “specific  difficult  goals  lead  to  better 
performance  than  ‘do  your  best’  or  easy  goals  and  that  goal  effects  are  moderated  by  feedback 
and  goal  commitment”  (p.  415).  Locke  and  Bryan  (1966)  found  support  for  this  theory.  They 
found  that  the  performance  and  intensity  of  effort  was  higher  for  those  given  specific  goals  when 
compared  to  those  told  to  “do  your  best.”  Thus,  a  clearly  stated  instructional  objective  should 
help  to  motivate  the  learner  by  making  the  learning  goals  more  specific. 
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Rothkopf  and  Billington  (1979)  compared  the  reading  times  and  eye  movements  in  two 
groups  of  readers.  One  group  was  given  goals  prior  to  reading  and  the  other  group  was  not  given 
goals.  They  found  that  readers  who  were  given  goals  spent  less  overall  time  reading,  but  more 
time  on  the  goal-relevant  information.  They  also  showed  better  recall  of  the  goal-specific 
information  than  background  information.  Similar  results  were  reported  in  a  comparison  of 
learning  from  computer  lessons  that  either  included  or  excluded  learning  objectives  (Cavalier  & 
Klein,  1998). 

Catrambone  (1998)  conducted  four  experiments  that  demonstrated  improved  problem-solving 
performance  when  subjects  were  given  labels  for  groups  of  steps  that  formed  subgoals.  Such 
subgoals  are  similar  to  separate  instructional  objectives  that  support  the  overall  purpose  of  the 
course.  When  learners  are  shown  the  relevance  of  the  instructional  objectives,  they  should  learn 
better. 

Clearly  stated  instructional  objectives  also  help  the  instructional  developers.  Merrill  (1983) 
recommends  that  objective  statements  should  include  “a  classification  of  the  type  of  learning 
involved;  affective,  psychomotor,  intellectual  skills,  verbal  information,  or  cognitive  strategies” 
(p.  290).  The  purpose  of  this  classification  is  to  help  instructional  developers  determine  the 
necessary  conditions  for  learning. 

A  complete  instructional  objective  should  meet  the  following  requirements  (Trollinger,  2003, 
citing  Dick  and  Carey,  1996;  Clark,  D.,  1995b;  Nash,  2001;  and  Department  of  Defense,  2001): 

•  A  statement  identifying  the  learners 

•  A  statement  of  what  the  learners  will  be  able  to  do  (their  capabilities) 

•  A  description  of  the  context  (conditions)  in  which  the  skills  will  be  applied 

•  A  description  of  the  tools  that  will  be  available  to  the  learners 

•  A  statement  of  how  the  learner  will  achieve  the  objective 

•  How  the  objective  relates  to  other  objectives  in  the  course 

As  can  be  seen  from  the  above  definitions,  an  instructional  objective  contains  three  main 
parts:  1)  a  description  of  an  observable  action  in  terms  of  performance  or  behavior;  2)  at  least 
one  measurable  criterion  (standard)  that  states  the  level  of  acceptable  performance  in  terms  of 
quantity,  quality,  time,  or  other  measures;  and  3)  the  actual  condition(s)  under  which  the  task 
will  be  observed. 

Observable  Action.  The  action  or  behavior  states  what  a  learner  will  do  to  demonstrate  that 
he  or  she  has  learned  what  is  required.  It  must  include  an  action  verb  such  as  “ type  a  letter,”  “ lift 
a  load,”  or  “ select  a  menu  option.”  Each  objective  should  cover  only  one  behavior  and  should 
include  only  one  verb.  If  multiple  behaviors  or  complicated  behaviors  are  required,  the  objective 
should  be  broken  down  into  enabling  objectives  that  support  the  main  terminal  objective. 
Department  of  Defense  (2001)  provides  a  table  of  standardized  action  verbs  for  instructional 
objectives  (Appendix  C). 

Measurable  Criterion  (standard).  The  standard  of  performance  answers  questions  such  as 
“How  many?”  “How  fast?”  or  “How  well?”  A  single  instructional  objective  can  contain  more 
than  one  measurable  criterion  depending  on  the  complexity  of  the  task.  However,  as  Don  Clark 
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(1995b)  cautions,  one  should  not  fall  into  the  trap  of  using  only  a  time  constraint  because  it  is  not 
easy  to  find  another  measure  of  performance.  A  time  limit  should  only  be  used  when  it  is 
required  under  normal  working  conditions. 

Conditions  of  Performance.  In  addition  to  stating  the  conditions  under  which  the  task  will 
occur,  this  portion  of  an  instructional  objective  identifies  the  tools,  procedures,  materials,  aids,  or 
facilities  to  be  used  in  performing  the  task.  It  is  often  expressed  with  a  prepositional  phrase  such 
as  “without  reference  to  the  manual”  or  “using  a  volt-ohm  meter.” 


Example  Instructional  Objectives.  Don  Clark  (1995b)  provides  four  examples  of 
instructional  objectives  (p.  5-6  of  22).  They  are  shown  in  Table  7. 


Table  7: 

Four  Example  Instructional  Objectives 


Example  1:  Write  a  customer  reply  letter  with  no  spelling  mistakes  by  using  a  word  processor. 

•  Observable  Action:  write  a  customer  reply  letter 

•  Measurable  Criteria:  with  no  spelling  mistakes 

•  Conditions  of  Performance:  using  a  word  processor 

Note:  If  more  than  one  type  of  word  processor  or  computer  is  used  in  the  organization,  the  objective  should  be 
more  specific  (e.g.,  Given  a  personal  computer,  Word  for  Windows,  and  a  printer,  create  a  printed  customer  reply 
letter  with  no  spelling  mistakes.) 

In  general,  the  larger  the  organization  and  the  more  technical  the  task,  the  more  specific  the  conditions  should  be 

specified. _ _ 

Example  2:  Copy  a  table  from  a  spreadsheet  into  a  word  processor  document  within  3  minutes 
without  reference  to  the  manual. 

•  Observable  Action:  copy  a  table  from  a  spreadsheet  into  a  word  processor  document 

•  Measurable  Criteria:  within  3  minutes 

•  Conditions  of  Performance:  without  referencing  the  manual _ 

Example  3:  Smile  at  all  customers,  even  when  exhausted,  unless  the  customer  is  irate. 

Note:  This  example  includes  a  variable. 

•  Observable  Action:  smile 

•  Measurable  Criteria:  at  all  customers 

•  Conditions:  even  when  exhausted 

•  Variable:  unless  the  customer  is  irate _ 

Example  4:  After  training,  the  worker  will  be  able  to  load  a  dumptruck  within  3  loads  with  a 
scooploader,  in  the  hours  of  darkness,  unless  the  work  area  is  muddy. 

•  Observable  Action:  load  a  dumptruck 

•  Measurable  Criteria:  within  3  loads 

•  Conditions:  with  a  scooploader  in  the  hours  of  darkness 

•  Variable:  unless  the  work  area  is  muddy _ 
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l.d.  The  Content  Supports  Each  and  Every  Instructional  Objective 

Once  the  learner  has  been  told  the  instructional  objectives  of  a  course  segment,  the  content 
should  provide  all  the  necessary  information  and  instruction  to  allow  the  learner  to  meet  each 
objective  (Nash,  2003). 

Using  the  Scale 

This  scale  is  used  to  rate  the  course  in  terms  of  how  many  instructional  objectives  are 
supported  by  the  course  content.  Evaluators  should  refer  to  the  anchor  statements  below  the 
scale  to  help  them  assign  a  score  on  this  criterion.  Additional  information  to  help  the  evaluator 
assign  a  score  is  provided  in  the  discussion  below.  Remember,  if  you  assigned  a  score  of  one  (1) 
on  Criterion  l.c,  you  must  also  assign  a  score  of  one  (1)  on  this  criterion. 

Terms.  Concepts,  Techniques,  and  Supporting  Literature 

Determining  if  the  content  supports  each  instructional  objective  is  easier  if  the  objectives  are 
clearly  stated  (l.c).  However,  if  an  objective  is  stated,  but  has  no  supporting  content,  it  cannot 
be  learned.  On  the  other  hand,  the  content  should  not  include  gratuitous  information  that  does 
not  support  the  instructional  objectives. 

The  consequences  of  not  meeting  the  instructional  objectives  or  purpose  of  the  course  are 
likely  a  loss  of  learner  motivation  and  poorer  performance  on  the  job.  The  data  on  the  effects  of 
“seductive  details”  are  relevant  here  (Gamer,  et  ah,  1989;  Harp  &  Mayer,  1998).  When 
information  is  introduced  that  is  not  relevant  to  the  instructional  objectives,  the  learner’s 
attention  may  be  diverted  to  this  irrelevant  information  and  may  never  be  refocused  on  the 
instructionally  important  information. 


l.e.  The  Content  is  Free  of  Errors 

Learners  will  have  more  faith  in  the  quality  of  the  course  and  probably  attend  more  closely  to 
the  instruction  if  they  can  see  that  the  content  is  free  of  errors.  If,  on  the  other  hand,  they 
observe  many  typographical  errors,  grammatical  errors,  mathematical  errors,  contradictory 
definitions,  or  errors  in  the  organization  of  the  course  (e.g.,  sections  that  are  numbered 
incorrectly),  they  may  suspect  that  the  course  developers  were  equally  careless  in  terms  of  the 
factual  accuracy  of  the  content. 

Using  the  Scale 

This  scale  is  used  to  evaluate  the  course  in  terms  of  how  many  factual  and/or  distraction 
errors  are  found  in  the  course.  The  evaluator  should  use  the  anchor  statements  below  the  scale  to 
help  determine  the  score  to  assign  on  this  criterion.  A  list  of  types  of  factual  and  distraction 
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errors  is  also  included  below  the  scale.  Additional  guidance  to  help  the  evaluator  is  provided  in 
the  discussion  below. 

Terms.  Concepts.  Techniques,  and  Supporting  Literature 

Content  errors  can  occur  in  two  forms:  factual  errors  and  distraction  errors.  Factual  errors  are 
those  that  are  commonly  called  “mistakes.”  They  can  include  mistakes  such  as: 

•  Incorrect  definitions 

•  Incorrect  formulas 

•  Mathematical  errors 

•  Logical  errors  in  examples  or  presentations 

•  Incorrect  numbering  of  course  segments 

•  Feedback  that  identifies  an  incorrect  answer  as  correct 

•  Other  obvious  factual  mistakes 

Related  to  factual  errors  are  distraction  errors  that  misdirect  the  attention  of  the  learner  from  the 
important  instructional  information  in  the  content.  Factual  errors  can  also  be  distractions,  but  are 
usually  easier  to  locate  than  other  types  of  distraction  errors.  Distraction  errors  are  particularly 
troublesome  because  the  learner  may  not  be  aware  that  he  or  she  is  distracted.  However,  if 
learners  are  thinking  about  something  other  than  the  instructional  information,  they  will  probably 
learn  less  effectively.  Distraction  errors  can  include: 

•  Typographical  errors 

•  Mislabeled  graphics 

•  Grammatical  errors 

•  Shifts  in  tense 

•  Shifts  in  voice  (e.g.,  human  voice  changes  to  computer-generated  voice  or  human  voice 

changes  to  another  accent) 

•  Unexpected  addition  of  sounds  or  narration,  such  as  one  page  of  a  web-delivered 

course  suddenly  having  narration  when  the  rest  of  the  pages  had  no  narration  (also  the 

reverse) 

•  Shifts  in  sound  level 

•  Unexpected  changes  in  control  locations  (see  section  5.f) 

•  Malfunctions  in  interface  controls  (see  section  5.g.) 

•  Other  obvious  distractions 

There  are  a  number  of  ways  that  the  design  of  content  can  increase  a  learner’s  faith  in  its 
factual  accuracy.  First,  the  course  should  be  thoroughly  reviewed  to  minimize  or  eliminate 
obvious  errors.  The  textual  content  should  be  grammatically  correct  and  free  of  typographical 
errors.  Mistakes  in  definitions  should  be  eliminated  by  closely  reviewing  sources  to  ensure  they 
are  authoritative  (e.g.,  articles  from  professional  journals  or  widely  used  guidebooks  or 
instructions).  Numerical  calculations  should  be  checked  for  accuracy  and  illustrative  examples 
should  be  thoroughly  reviewed  by  subject  matter  experts  (SMEs)  for  both  factual  and  logical 
errors.  Care  should  be  taken  to  ensure  the  reliability  of  the  content  by  eliminating  internal 
contradictions  like  conflicting  definitions,  confusing  examples,  or  feedback  that  identifies 
incorrect  answers  as  correct  or  correct  answers  as  incorrect.  In  addition,  a  glossary  containing 
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the  definitions  of  all  important  terms  should  also  be  provided  and  used  as  the  baseline  for  review 
of  terms  throughout  the  course. 

Some  of  the  empirical  support  for  this  criterion  is  found  in  the  literature  on  selective  attention 
(Broadbent,  1963;  Klatsky,  1975;  Norman,  1976).  Errors  in  the  course  can  become  distracters, 
leading  the  learners’  attention  away  from  the  most  important  information.  This  is  similar  to  the 
effect  of  “seductive  details”  (Gamer,  et  al,  1989;  Harp  &  Mayer,  1998).  It  is  especially  critical 
to  avoid  errors  in  asynchronous  courses  which  do  not  have  an  instructor  to  monitor  and  redirect 
the  learners’  attention. 

It  is  also  important  to  minimize  errors  in  introductory  courses.  Novices  are  not  as  familiar 
with  the  nuances  of  a  topic  as  experts  (Wickens  &  Hollands,  2000,  p.  255).  The  novices  may 
become  even  more  distracted  because  they  don’t  know  which  errors  are  important  and  which  are 
only  nuisances. 


l.f.  The  Content  is  Job-relevant 


This  scale  is  used  to  evaluate  the  course  in  terms  of  the  amount  of  content  that  is  relevant  to 
the  learner’s  job. 

Using  the  Scale 

Anchor  statements  and  a  list  of  ways  to  make  content  job-relevant  are  found  below  the  scale. 
The  evaluator  should  use  these  to  help  assign  a  score  on  this  criterion.  Additional  information  to 
help  the  evaluator  is  provided  in  the  discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

Often  people  are  “forced”  to  take  instruction  that  they  perceive  as  a  waste  of  time  (e.g., 
“sensitivity”  training,  “risk  management”  training).  This  may  be  a  reason  why  transfer  of 
learning  to  job  performance  is  often  disappointing.  “Most  of  the  research  on  employee  training 
clearly  shows  that  although  millions  of  dollars  are  spent  on  training  in  the  public  sector,  there  is 
little  empirical  evidence  linking  training  to  improved  job  behavior  or  employee  attitudes” 
(Haskell,  2001,  p.  5  cited  in  Clark,  R.  C.,  2003,  p.  136). 

Educational  researchers  have  looked  at  the  differences  between  experts  and  novices  in  how 
they  organize  important  knowledge  and  skills  in  a  domain.  It  has  been  shown  that  experts  have 
developed  a  more  complex  model  of  a  topic  area  and  are  able  to  attend  to  more  subtle  features  of 
the  topic  when  solving  problems  (e.g.,  Chi,  Feltovich,  &  Glasser,  1981).  Bransford  and  his 
colleagues  (Bransford,  Sherwood,  Vye,  &  Rieser,  1986;  Bransford,  Franks,  Vye,  &  Sherwood, 
1989)  argue  that  instruction  that  focuses  only  on  facts  does  not  guarantee  that  the  learner  will  be 
able  to  use  that  knowledge  in  later  situations.  “Our  data  suggest  that  fact-oriented  acquisition 
permits  people  to  remember  information  when  explicitly  prompted  to  do  so,  but  it  does  not 
facilitate  their  ability  to  use  this  information  spontaneously  to  solve  new  problems”  (Bransford, 
et  al.,  1989,  p.  479).  They  recommend  that  instruction  should  provide  information  in  the  context 
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of  solving  relevant  problems.  Furthermore,  they  found  that  problem-oriented  activities  lead  to 
“greater  spontaneous  use  of  the  acquired  information  to  aid  in  problem  solving”  (p.  477). 

The  relevance  of  the  instructional  material  affects  the  learner’s  attention  (Broadbent,  1963; 
Klatsky,  1975;  Norman,  1976),  motivation  (Locke  &  Bryan,  1966;  Magill,  1980;  Ausubel, 

1968),  and  depth  of  processing  of  the  content  (Brown  &  McNeill,  1966;  Craik  &  Lockhart,  1972; 
Anderson  &  Bower;  1972;  1973;  1974).  Therefore,  in  addition  to  stating  the  purpose  of  the 
instruction  (La)  and  what  the  learner  should  be  able  to  do  after  the  instruction  (Lb),  the  learner 
should  be  able  to  relate  the  content  to  requirements  of  his  or  her  job  (i.e.,  it  should  be  job¬ 
relevant).  This  means  learners  should  be  able  to  readily  perceive  that  the  content  is  meaningful 
in  the  context  of  the  job  and  that  it  supports  their  job  requirements  (Dick  and  Carrey,  1996; 

Texas  Education  Agency,  2003;  Trollinger,  2003).  As  discussed  under  Criterion  La,  Nitsch 
(1977)  and  Bransford  and  his  colleagues  (1986, 1989)  believe  that  the  instruction  on  a  topic 
should  be  structured  so  that  learners  can  establish  a  firm  contextual  basis  for  understanding  the 
new  concept.  This  contextual  basis  should  include  concrete,  job-relevant  examples. 

The  content  should  be  free  of  jargon  and  only  use  terms  that  are  defined  and  job-relevant. 
Some  of  the  techniques  that  can  make  content  more  job-relevant  include:  use  of  real-world 
examples  from  the  job  area,  testimonials  from  experts  in  the  field,  or  use  of  realistic  scenarios. 

The  Texas  Education  Agency  (2003)  recommends  that  learners  be  given  the  ability  to 
“customize”  the  instruction  to  meet  their  specific  needs.  However,  Jones  and  Okey  (1995)  citing 
Jones  (1989)  caution  that  “browsing”  through  a  course  should  not  be  indiscriminate  or 
uncontrolled.  Rather,  they  recommend  that  content  be  designed  and  organized  so  that  all 
learners  are  given  instruction  on  “required”  topics  (e.g.,  those  that  everyone  in  an  occupational 
field  must  master).  Then  additional,  related,  detailed  content  could  be  presented  in  a  manner  that 
allows  them  to  “drill  down”  into  this  content  through  more  and  more  levels  of  information  to  the 
appropriate  level  of  the  topics  that  is  most  relevant  to  their  needs.  This  can  be  done  using  pop¬ 
up  menus,  buttons,  hot  text,  or  other  methods. 


l.g.  The  “Authority”  for  the  Content  is  Clearly  Stated 

As  discussed  above  (l.e),  the  learner  needs  to  feel  confident  that  the  content  is  factually 
accurate.  Although  it  is  fairly  easy  for  the  learner  to  determine  if  the  course  is  free  of  surface 
errors,  it  is  more  difficult  for  a  learner  to  judge  the  factual  accuracy  of  the  content.  This  is 
especially  true  if  the  learner  is  a  novice  in  the  subject  area. 

Using  the  Scale 

This  scale  is  used  to  evaluate  the  quantity  and  quality  of  references  and  credits  that  are 
provided  to  substantiate  the  “authority”  for  the  content.  The  evaluator  should  use  the  anchor 
statements  below  the  scale  to  help  assign  a  score  on  this  criterion.  Additional  information  to 
assist  the  evaluator  is  provided  in  the  discussion  below. 
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Terms,  Concepts.  Techniques,  and  Supporting  Literature 

The  content  presented  in  an  instructional  product  consists  of  a  number  of  factual  assertions  or 
claims.  The  justification  for  these  assertions  relies  on  their  support  evidence  or  the  authority 
behind  the  claims.  Zarefsky  (2001)  discusses  several  types  of  evidence  that  can  support  factual 
assertions.  One  type  of  justification  is  social  consensus  or  common  knowledge.  Secondly,  they 
can  be  justified  by  objective  data  (e.g.,  examples,  statistics,  tangible  evidence,  or  historical 
documents).  Finally,  the  credibility  of  the  information  can  be  judged  by  the  credibility  of  the 
source  of  the  information.  That  is,  the  competence  and  trustworthiness  of  the  source. 

Social  consensus  may  appear  straightforward,  with  statements  like,  “Everyone  knows  such 
and  such”  perceived  as  the  only  justification  necessary.  However,  one  must  be  aware  that 
common  knowledge  may  not  always  be  correct  knowledge.  It  is  always  better  to  supplement 
social  consensus  with  other  justifications.  Indeed,  Bell  (2005)  cautions  that  statements  like 
“everyone  knows  such  and  such”  are  propaganda  techniques.  He  believes  that  critical  thinkers 
should  not  fall  into  the  trap  of  believing  such  statements. 

In  terms  of  objective  data,  one  must  look  to  the  methods  used  to  collect  the  data  and  the  logic 
behind  its  interpretation.  In  most  cases,  an  evaluator  or  a  learner  does  not  have  the  expertise  to 
judge  whether  these  methods  were  correctly  employed.  Therefore,  one  must  rely  on  “experts,” 
such  as  review  boards  or  individuals  with  experience  in  the  field.  In  these  cases,  it  is  the 
credibility  of  the  source  that  is  actually  being  used  as  justification. 

There  are  several  questions  that  one  might  ask  to  reduce  doubts  about  the  credibility  of  an 
information  source.  First,  one  could  ask,  “Is  the  person  an  authority  in  the  field?”  This  can  be 
supported  if  the  individual  has  made  the  factual  assertions  in  respected  forums  (e.g.,  scientific 
journals,  books,  professional  meetings).  Second,  one  could  ask,  “Does  the  person  have  a  vested 
interest?”  Eagly,  Wood,  and  Chaiken  (1978)  found  that  communicators  were  judged  as  being 
less  biased  and  more  persuasive  when  they  made  arguments  against  their  own  self-interest.  If 
the  communicator  appears  unbiased  and  unselfish,  the  message  is  likely  to  be  perceived  to  be 
based  on  compelling  evidence.  Credibility  can  also  be  judged  by  asking  a  third  question,  “Do 
other  credible  sources  disagree?”  These  are  the  types  of  questions  that  should  be  asked  by 
instructional  developers  when  assembling  the  content  for  a  course. 

Learners  do  not  judge  the  credibility  of  a  course  by  asking  specific  questions  like  those 
above.  Perceptions  of  source  credibility  are  easily  manipulated.  For  example,  Miller, 

Moruyama,  Beaber,  and  Valone  (1976)  found  that  listeners  judged  speakers  as  more  credible  if 
they  spoke  at  a  rapid  rate.  This  result  should  not  be  justification  for  rapid  narrations.  Rather,  the 
findings  emphasize  the  importance  of  considering  the  listener’s  perception  when  evaluating 
“authority.” 

Learner  confidence  in  the  authority  of  content  can  be  expected  to  increase  if  the  origin  of  the 
content  is  made  apparent  as  well  as  the  qualifications  of  the  individuals  who  provided  the 
content  (the  subject  matter  experts)  and  created  the  instruction  (the  instructional  developers). 

This  is  even  true  for  the  reviewers  of  the  course  (e.g.,  “an  instructional  quality  review  was 
conducted  by. . .”).  This  information  should  be  made  available  to  the  learners  in  a  bibliography 
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and  collection  of  resumes  and  summary  statements  that  are  easily  accessed  from  the  course. 
Furthermore,  any  disclaimers  or  cautions  about  the  content  (e.g.,  disagreements  among  experts, 
theories  versus  facts,  etc.)  should  be  provided  to  the  learners  (Bostock,  2004). 

Another  way  to  increase  learner  confidence  is  to  ensure  that  the  content  is  current  (e.g., 
includes  dates  for  all  references  and  the  version  of  the  course).  The  latest  references,  as  well  as 
“classic”  references  should  be  included  and  updated  periodically.  Periodic  reviews  of  examples, 
simulations,  and  testimonials  should  also  be  conducted  to  ensure  that  they  are  providing  the  most 
up-to-date  information  on  the  topic. 

l.h.  There  are  Clear  Indications  of  Prerequisites 

As  discussed  in  Section  1  .a,  the  structure  of  instructional  information  can  positively  or 
negatively  affect  the  learning  experience.  Gagne  (1968;  1985)  strongly  argued  that  instruction 
should  be  built  as  a  hierarchy  of  information.  This  also  is  the  basis  for  the  Elaboration  Theory  of 
Instruction  (Reigeluth  &  Stein,  1983).  This  hierarchy  should  be  readily  apparent  to  the  learner. 
One  way  to  help  ensure  that  the  learner  is  attending  to  the  appropriate  level  of  this  informational 
hierarchy  is  to  clearly  indicate  the  prerequisites  for  any  course  or  course  segment.  The 
indications  of  prerequisites  serve  as  guideposts  for  the  learner  and  help  to  ensure  that  he  or  she  is 
taking  the  appropriate  segment  of  instruction. 

Using  the  Scale 

This  scale  is  used  to  evaluate  the  course  in  terms  of  how  many  of  the  needed  prerequisites  are 
clearly  indicated.  Evaluators  should  use  the  anchor  statements  below  the  scale  and  the  list  of 
ways  to  indicate  prerequisites  to  help  them  assign  a  score  on  this  criterion.  Additional 
information  on  this  criterion  is  provided  in  the  discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

As  previously  discussed  (l.a),  learners  should  proceed  through  a  course  in  small  meaningful 
steps  (Gagne,  1968).  Before  beginning  each  step,  the  learner  should  have  completed  all  required 
prerequisite  topics.  Catrambone  (1998)  demonstrated  that  labeling  groups  of  steps  in  examples 
helped  learners  understand  these  meaningful  steps  or  subgoals.  Royer  (1979)  reviewed  several 
theories  of  transfer  and  concluded  that  “relevant  previous  learning  can  facilitate  the  acquisition 
of  current  instruction”  (p.  53).  Prerequisites  provide  the  essential  information  that  is  required  by 
the  learner  to  organize  and  structure  new  instructional  information. 

In  an  introductory  course,  the  learners  should  not  have  to  worry  about  prerequisites  because 
the  course  should  be  designed  to  guide  them  through  the  content  in  the  appropriate  sequence. 
However,  learners  could  overlook  important  prerequisites  in  a  more  advanced  course.  This 
could  lead  to  confusion  and  frustration  when  they  have  not  learned  this  prerequisite  knowledge 
or  mastered  these  necessary  skills.  In  all  cases,  the  introduction  or  course  overview  should 
provide  clear  statements  about  requirements  for  prerequisite  courses.  Within  a  course,  each 
topic  should  include  an  introductory  discussion  about  prerequisite  topics. 
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l.i.  There  are  Clear  Indications  of  Completed  Topics 

Learners  should  always  know  when  they  have  completed  a  topic.  They  should  not  have  to 
wonder  whether  new  information  is  part  of  the  current  topic  or  the  beginning  of  a  new  topic.  A 
clear  indication  of  topic  completion  enables  the  learner  to  cognitively  pause  and  consolidate  the 
information  presented. 

Using  the  Scale 

This  scale  is  used  to  rate  the  course  on  how  clearly  the  completion  of  topics  are  indicated. 
Evaluators  should  use  the  anchor  statements  below  the  scale  and  the  list  of  indications  of 
completed  topics  to  help  them  assign  a  score  on  this  criterion.  Additional  information  on  this 
criterion  is  provided  in  the  discussion  below. 

Terms,  Concepts.  Techniques,  and  Supporting  Literature 

A  learner  should  not  have  to  wonder  if  he  or  she  has  completed  a  topic.  The  structure  of  the 
course  should  provide  the  cues  that  the  learner  completed  a  topic.  Some  of  these  structural  cues 
include  summaries  of  the  important  information,  interactive  activities  (2.a),  or  some  form  of 
assessment  (Chapter  5,  Evaluation  Area  3)  to  determine  if  the  learner  has  learned  the  material 
and  is  ready  to  move  forward.  Performance  feedback  (Chapter  6,  Evaluation  Area  4)  should  then 
be  provided  to  either  reinforce  correct  behaviors  or  to  provide  remedial  instruction  if  the  learner 
has  not  demonstrated  that  the  material  has  been  learned  correctly.  In  all  cases,  the  learner  should 
be  provided  with  an  unambiguous  message  (or  multiple  messages  using  various  techniques)  that 
the  topic  has  been  completed.  For  example,  a  topic  on  the  table  of  contents  could  change  color 
when  it  has  been  successfully  completed. 


l.i.  Sources  for  Additional  Information  are  Available 


Every  learner  does  not  take  a  course  for  the  same  reason.  The  requirements  of  a  learner’s  job 
may  lead  him  or  her  to  desire  or  need  additional  information  on  the  topics  presented  in  the 
course.  Sources  for  this  additional  information  should  be  made  available  and  instructions 
provided  to  access  these  sources. 

Using  the  Scale 

This  scale  is  used  to  evaluate  the  course  in  terms  of  how  well  the  course  provides  sources  for 
additional  information.  Evaluators  should  use  the  anchor  statements  below  the  scale  to  help 
them  assign  a  score  on  this  criterion.  Additional  information  on  this  criterion  is  provided  in  the 
discussion  below. 


82 


Technical  Report  2005-002 


Terms.  Concepts.  Techniques,  and  Empirical  Support 

No  course  can  provide  all  information  on  any  given  topic  nor  is  this  a  goal  that  should  be 
pursued.  The  goal  of  instruction  should  be  to  achieve  the  purposes  and  instructional  objectives 
of  the  course.  Many  learners  will  wish  to  learn  more  about  some  of  the  topics  in  the  course. 
Some  of  this  additional  instructional  information  can  be  provided  as  “drill  down”  options  in  the 
course.  However,  most  additional  information  will  be  found  in  other  sources.  The  course  should 
provide  easy-to-follow  instructions  or  directions  to  find  this  information.  Some  of  the  sources 
might  be  additional  courses,  literature,  professional  organizations,  schedules  for  meetings,  etc. 

In  all  cases,  the  developers  and  maintainers  of  the  course  should  ensure  that  the  information  from 
the  sources  is  current,  the  directions  to  the  sources  are  correct,  and  that  the  sources  are  still 
available. 
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Chapter  4: 

Instructional  Activities 
(Evaluation  Area  2) 

“Time  plus  energy  equals  learning”  (Chickering  &  Gamson,  quoted  in  Graham,  Cagiltay, 
Craner,  Byung-Ro,  &  Duffy,  2000).  This  short  phrase  captures  one  of  the  most  important 
concepts  of  instruction.  The  learner  must  invest  time  and  energy  in  order  to  learn.  He  or  she 
should  not  be  a  passive  recipient  of  instruction.  Rather,  the  instructional  product  should  be 
designed  so  the  learner  is  an  active  participate  in  the  learning  process  and  is  encouraged  and 
enabled  to  think  more  about  the  subject  or  practice  the  required  skills.  This  chapter  provides 
discussions  of  the  criteria  that  deal  with  what  the  learner  does  during  the  course. 

2.a.  Activities  are  Relevant  (All  Support  LOs  and  Job  Requirements 

Activities  should  be  designed  to  support  the  learning  objectives  (LOs)  of  the  course.  The 
instructional  activities  should  provide  the  learners  with  the  necessary  skills,  knowledge,  and 
experience  to  meet  the  goals  and  objectives  of  the  course  and,  ultimately,  their  job. 

Using  the  Scale 

This  scale  is  used  to  rate  how  many  instructional  activities  included  in  the  course  are  relevant. 
Use  the  anchor  statements  below  the  scale  and  the  examples  of  relevant  activities  to  help  assign  a 
score  on  this  criterion.  Remember,  if  you  assign  a  rating  of  one  (1)  on  this  criterion,  do  not 
continue.  The  course  needs  to  be  reworked.  Additional  information  to  help  the  evaluator 
understand  the  importance  of  relevant  activities  is  provided  in  the  discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

According  to  Rumelhart  and  Norman  (1981,  p.  358),  analogical  reasoning  is  the  most 
common  way  that  people  apply  knowledge  learned  in  one  domain  to  another  (e.g.,  from  one  job 
task  to  another  or  from  an  instructional  situation  to  a  job  situation).  Instructional  activities 
should  be  designed  to  help  the  learner  develop  analogical  reasoning  by  exposing  the  learner  to 
natural,  relevant  operations  and  information  from  the  target  domain. 

The  learner  should  find  the  instructional  activities  meaningful.  That  is,  the  learner  should  be 
able  to  easily  see  how  the  learning  objectives  are  supported  by  the  activities  (Perry,  2003).  The 
learner  should  also  easily  perceive  that  the  activities  are  job-relevant  (i.e.,  they  have  face 
validity).  Some  examples  of  relevant  activities  include: 

•  Role  playing  in  a  case  study  based  on  real-world  issues 

•  Solving  realistic  (real-world)  problems 

•  Interacting  with  realistic  simulations 

•  Identifying  the  similarities  and  differences  among  real-world  examples 

•  Using  analogies 

Information  on  these  activities  is  provided  in  the  sections  below. 
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Role  Playing 

Role  playing  is  a  type  of  activity  which  allows  the  learner  to  put  him  or  herself  into  a  realistic 
situation  when  he  or  she  may  use  the  information  he  or  she  is  trying  to  learn.  Taylor,  Renshaw, 
and  Jensen  (1997)  assessed  the  use  of  role  playing  exercises  in  either  a  paper-and-pencil  or 
computer-based  format  for  learning  information  about  the  probability  of  a  volcano  eruption. 

They  also  manipulated  whether  the  role  playing  scenarios  were  framed  positively  (e.g.,  it  is 
estimated  that  no  eruption  will  occur)  or  negatively  (e.g.,  it  is  estimated  that  an  eruption  will 
occur).  Results  indicated  that  computer-based  role  playing  enhanced  learning.  Furthermore,  in 
the  computer-based  condition,  there  was  no  effect  of  the  type  of  framing.  These  results  should 
be  regarded  with  caution  because  of  differences  between  the  computer-based  and  paper-and- 
pencil  exercises  that  could  also  have  affected  the  results  (e.g.,  novelty,  timing  of  feedback,  use  of 
animation,  etc.). 

Problem  Solving 

Mayer,  Dow,  and  Mayer  (2003)  conducted  a  series  of  four  experiments  to  examine  some  of 
the  variables  that  affect  learner  performance  in  a  problem-solving  task.  In  all  the  experiments, 
the  learners  were  presented  with  explanations  from  an  agent  (Dr.  Phyz).  In  the  first  experiment, 
the  learners  were  given  explanations  on  the  operation  of  electric  motors  by  either  an  on-screen 
agent  or  as  on-screen  text.  The  group  receiving  narration  performed  better  than  the  text  group  on 
the  problem-solving  transfer  test. 

In  experiment  2,  the  role  of  interactivity  was  examined.  Learners  were  presented  with 
explanation  of  lightning  formation  and  they  either  were  allowed  to  control  the  pace  of  the 
explanations  or  were  presented  with  non-interactive  messages  that  were  not  under  their  control. 
The  learners  in  the  interactive  group  generated  significantly  more  answers  on  the  problem¬ 
solving  test  than  the  non-interactive  group.  The  same  procedure  was  tried  again  but  the  transfer 
test  was  administered  one  week  after  the  learning  session.  In  this  case,  the  interactive  group  also 
outperformed  the  non-interactive  group. 

In  their  third  experiment,  half  of  the  learners  were  given  pre-questions  to  guide  their  learning 
of  electric  motors.  This  group  performed  better  than  the  group  that  did  not  receive  the  pre¬ 
questions.  The  researchers  assumed  that  the  pre-questions  required  the  learners  to  more  deeply 
process  the  information  and  to  think  about  the  information  prior  to  and  during  the  learning 
session.  The  results  provide  support  for  the  self-explanations  principle  (people  learn  better  when 
they  are  prompted  to  provide  explanations  during  learning). 

Finally,  in  their  fourth  experiment,  they  examined  the  role  of  the  agent’s  on-screen  image. 

No  performance  differences  were  found  between  the  group  who  learned  with  an  image  of  the 
agent  present  during  narration  and  the  group  who  were  only  given  narration  without  an  image  of 
the  agent.  The  results  indicate  that  the  presence  of  an  agent  may  not  always  enhance  learning 
and  may  even  be  distracting.  This  is  probably  affected  by  the  nature  of  the  task  and  the 
characteristics  of  the  agent.  Indeed,  Stout  and  Sims  (2005)  argued  the  presence  of  agents  may  be 
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beneficial  for  instruction  on  some  types  of  tasks  (e.g.,  the  need  for  learners  to  be  able  to  react  to 
facial  expressions  or  body  movements  in  specific  ways). 

Simulations 


Instructional  simulations  are  usually  created  by  combining  a  variety  of  media  attributes  (e.g., 
sound,  motion,  and  text).  The  term  simulation  should  not  be  confused  with  simulator.  A 
simulator  is  a  type  of  device  which  “(a)  attempts  to  duplicate  the  essential  features  of  a  task 
situation  and  (b)  provides  for  direct  practice”  (Kinkade  &  Wheaton,  1972,  p.  671).  A  simulation, 
on  the  other  hand,  does  not  have  to  be  any  type  of  equipment  (although  it  may  involve  the  use  of 
equipment).  Morris  and  Thomas  (1976)  define  a  simulation  as  “the  ongoing  representation  of 
certain  features  of  a  real  situation  to  achieve  some  specific  training  objective”  (p.  66).  They  go 
on  to  define  a  simulator  as  “the  media  through  which  a  trainee  may  experience  the  simulation” 

(p.  66). 

The  Department  of  Defense  (1997)  identifies  three  types  of  simulations:  live,  virtual,  and 
constructive. 

•  Live  Simulations  involve  real  people  operating  real  systems.  An  example  is  the  Multiple 
Integrated  Laser  Engagement  Simulation  (MILES).  This  system,  the  precursor  of  the  laser 
tag  game,  attaches  laser  emitters  and  receivers  to  weapons  so  troops  can  practice  in  realistic 
conditions. 

•  Virtual  Simulations  involve  real  people  operating  simulated  systems.  A  flight  simulator  or 
a  driving  simulator  are  examples  of  virtual  simulations. 

•  Constructive  Simulations  involve  simulated  people  operating  simulated  systems.  Real 
people  make  inputs  into  the  simulation  and  the  simulation  determines  the  outcomes  of  the 
exercise.  Two  examples  are  a  computer-controlled  war  game  or  a  simulation  of  the 
activities  of  a  team  dealing  with  an  oil  spill. 

These  three  types  of  simulations  are  not  mutually  exclusive  and  are  often  combined  in  a  given 
instructional  exercise. 

The  use  of  simulations  has  been  an  effective  instructional  approach  for  training  people  on 
tasks  with  strong  perceptual-motor  components  such  as  flying  an  aiiplane  (Valverde,  1973; 

Waag,  1981;  Flaxman  &  Stark,  1987;  Hays  &  Singer,  1989;  Hays,  Jacobs,  Prince  &  Salas, 

1992).  Simulations  have  also  been  shown  to  be  effective  for  teaching  cognitive  concepts  like 
statistics.  Lane  and  Tang  (2000)  compared  the  performance  of  learners  given  a  simulation 
illustrating  the  statistical  concepts  of  probability  with  those  given  textual  explanations.  The 
learners  who  were  given  the  simulation  outperformed  the  learners  trained  with  a  textbook. 

Swaak  and  de  Jong  (2001)  studied  the  learning  effectiveness  of  simulations  for  teaching 
physics  principles.  They  found  that  the  simulations  enhanced  the  learning  of  the  principles. 
Furthermore,  they  found  that  activities  (exercises)  designed  to  help  the  learner  structure  the 
simulation  as  a  learning  experience  contributed  most  clearly  to  their  effectiveness. 

Determining  the  Characteristics  of  Simulations.  No  instructional  simulation  can  capture  all  of 
the  characteristics  of  a  real  situation.  If  it  did,  it  would  be  the  real  situation,  not  a  simulation. 
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Each  of  the  possible  characteristics  that  can  be  simulated  adds  to  the  cost  of  the  simulation  and 
increases  its  development  time.  Furthermore,  for  instructional  purposes,  it  is  often  desired  to 
omit  certain  characteristics  of  the  real  situation  so  the  instruction  can  focus  on  specific  tasks. 
Thus,  it  is  of  vital  importance  to  determine  which  characteristics  of  the  real  situation  can  and 
should  be  simulated.  A  conceptual  metaphor  that  can  help  simulation  designers  make  these 
important  choices  is  simulation  fidelity. 

Simulation  fidelity  is  the  degree  of  similarity  between  the  simulated  situation  and  the 
operational  situation  that  is  simulated.  This  degree  of  similarity  can  be  conceptualized  under 
two  major  dimensions:  (1)  the  physical  characteristics  of  the  simulation  (e.g.,  visual,  spatial, 
kinesthetic);  and  (2)  the  functional  characteristics  of  the  simulation  (e.g.,  the  informational 
(stimulus)  and  action  (response)  options  provided  by  the  simulation). 

Baum,  Reidel,  Hays,  and  Mirabella  (1982)  confirmed  the  separate  and  interactive  effects  of 
the  physical  and  functional  dimensions  of  fidelity  in  a  perceptual-motor  task.  They  concluded 
that  in  fidelity  research,  it  is  not  sufficient  to  study  general  levels  of  fidelity.  Rather,  fidelity 
must  be  operationalized  in  terms  of  at  least  two  dimensions:  physical  and  functional  similarity. 

Physical  fidelity  is  how  the  simulation  looks;  the  physical  characteristics  of  the  simulation. 
Functional  fidelity  is  how  the  simulation  works  or  provides  the  necessary  information  to  support 
the  task.  These  two  fidelity  dimensions  are  related  (e.g.,  if  the  physical  characteristics  of  the 
simulation  are  changed,  its  functional  fidelity  will  also  change).  In  general,  the  functional 
requirements  of  the  simulation  should  dictate  its  physical  characteristics. 

The  functional  fidelity  (the  information  or  stimulus  and  response  options)  is  how  the 
simulation  provides  required  information  and  interaction  opportunities  so  the  learner  can  learn 
the  task.  Depending  upon  the  task,  the  information  can  come  through  any  of  the  senses.  For 
example,  if  a  trainee  is  learning  to  fly  a  plane  in  a  simulated  cockpit,  and  the  task  is  to  fly  at  a 
specific  altitude,  then  the  simulated  altimeter  must  be  represented  in  enough  detail  that  the 
learner  can  read  the  altitude.  On  the  other  hand,  if  the  training  task  is  only  to  locate  the  display 
next  to  the  altitude  indicator,  then  the  simulation  does  not  need  to  include  a  functioning 
altimeter. 

In  order  to  respond  to  the  information  (act),  the  user  must  have  the  ability  to  make  input  to  the 
simulation.  For  example,  if  the  learner  needs  to  learn  how  to  change  altitude,  a  simulated  control 
device  must  actually  move  and  this  movement  must  result  in  a  change  in  the  simulated  altitude. 

The  instructional  developer  and  simulation  designer  must  ask  questions  like: 

•  “What  information  does  one  need  to  accomplish  the  task?” 

•  “How  does  one  obtain  this  information?” 

•  “What  actions  must  one  take  to  accomplish  the  task?” 

•  “How  does  one  take  these  actions  and  what  feedback  information  informs  one  that  the 
task  has  been  accomplished  correctly?” 

Answers  to  these  types  of  questions  help  determine  the  functional  requirements  of  the 
simulation.  These  functional  requirements  then  drive  the  decisions  about  how  to  design  the 
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physical  characteristics  of  the  simulation.  Additional  details  on  the  concept  of  fidelity  may  be 
found  in  Hays  and  Singer  (1989)  and  Hays  (in  prep). 

Working  with  Examples 

Using  examples  can  be  a  useful  technique  when  the  learning  task  involves  problem  solving 
and  the  learners  are  novices.  Ruth  Clark  defines  worked  examples  as  “a  step-by-step 
demonstration  of  how  to  perform  a  task  or  solve  a  problem”  (Clark,  R.  C.,  2003,  p.  51).  Paas 
(1992)  compared  the  effects  of  worked  examples  (practice  exercises)  on  learning  geometry 
problems.  One  group  was  given  conventional  instruction  using  12  practice  problems.  A  second 
group  studied  worked  examples  followed  by  four  practice  problems.  The  worked  examples 
group  spent  less  time  studying,  yet  scored  higher  on  a  test  than  did  those  in  the  conventional 
group.  The  worked  examples  group  also  scored  higher  on  different  types  of  problems  that 
required  applications  of  the  principles  taught.  Zhu  and  Simon  (1987)  found  that  worked 
examples  allowed  learners  to  finish  a  three-year  mathematics  course  in  two  years. 

It  should  be  noted  that  experienced  learners  do  not  appear  to  benefit  from  worked  examples. 
Kalyuga,  Chandler,  Tuovinen,  and  Sweller  (2001)  studied  trade  apprentices  learning  to  write 
programs  for  circuits.  In  the  early  sessions,  the  learners  benefited  from  the  worked  examples. 
However,  in  later  sessions,  solving  problems  was  found  to  be  a  more  effective  approach.  They 
concluded  that  novices  benefit  from  worked  examples,  but  those  with  more  experience  benefit 
just  as  well  from  less  structured  problem  solving  strategies  (see  also  Tuovien  &  Sweller,  1999). 
Based  on  their  research,  Kalyuga  et  al.  (2001)  provide  the  following  instructional  design 
recommendation:  “A  series  of  worked  examples  should  be  presented  first.  After  learners 
become  more  familiar  with  the  domain,  problem  solving  (as  well  as  exploratory  learning 
environments)  can  be  used  to  further  enhance  and  extend  acquired  skills”  (p.  588). 

Atkinson  (2002)  studied  the  instructional  effects  of  using  an  animated  pedagogical  agent  to 
deliver  instructional  information  about  worked  examples.  He  found  that  learners  exposed  to  an 
animated  narrator  outperformed  those  only  given  textual  explanations  of  the  examples.  Baylor 
and  Ryu  (2003)  studied  learners’  perceptions  of  pedagogical  agent  personal  characteristics. 

They  found  that  animated  agents  were  perceived  as  more  “instructor  like”  and  “engaging.” 
However,  the  learners’  perception  of  agent  credibility  was  found  to  be  the  same  with  either  a 
static  or  animated  agent. 

Moreno  and  Mayer  (2004)  examined  the  role  of  personalized  agent  messages  in  a  computer- 
based  simulation  used  to  teach  college  students  about  the  characteristics  of  plants.  They  found 
that  the  learners  who  received  personalized  agent  messages  performed  better  than  those  who 
received  non-personalized  messages  on  both  retention  and  problem-solving  transfer  tests.  This 
and  other  research  on  pedagogical  agents  has  implications  for  how  learners  interact  with  content 
(see  Criterion  2.b)  and  ways  to  make  the  content  more  enjoyable  (see  Criterion  2.c). 


89 


Technical  Report  2005-002 


Using  Analogies 

“An  instructional  analogy  is  an  explicit,  nonliteral  comparison  between  two  objects,  or  sets  of 
objects  that  describes  their  structural,  functional,  and/or  causal  similarities”  (Clark,  R.  C.,  2003, 
p.  111).  Analogies  can  help  the  learner  understand  instructional  information  by  relating  the  new 
information  to  familiar  knowledge.  For  example,  Newby,  Ertmer,  and  Stepich  (1995)  found 
positive  learning  benefits  from  the  use  of  analogies  when  teaching  physiological  concepts. 

Stepich  and  Newby  (1988)  maintain  that  there  are  four  parts  that  make  up  an  analogy: 

1 .  A  subject  which  is  the  new  information  to  be  learned  (e.g.,  the  functions  of  red  blood 
cells). 

2.  An  analog :  some  existing  knowledge  that  can  be  compared  to  the  subject  (e.g.,  trucks). 

3.  A  connector ,  which  shows  the  similarity  between  the  subject  and  the  analog  (e.g.,  red 
blood  cells  function  like  trucks...). 

4.  A  ground,  which  describes  the  similarities  and  differences  between  the  subject  and  the 
analog  (e.g.,  a  common  function  between  red  blood  cells  and  trucks  is  they  are  both 
transporters  of  materials;  a  difference  is  that  blood  cells  are  biological  and  trucks  are 
mechanical). 

Stepich  and  Newby  (1988)  also  provide  guidelines  for  constructing  and  presenting  analogies. 
When  constructing  an  analogy,  they  recommend  following  the  memory  aid  ABCD: 

•  A:  Analyze  the  subject  to  determine  the  most  important  things  the  learner  needs  to 
understand  and  the  characteristics  that  make  the  subject  unique  or  novel. 

•  B:  Brainstorm  potential  analogs  and  list  the  concrete  items  that  have  the  same  or  similar 
attributes  as  the  subject. 

•  C:  Choose  an  analog.  From  the  list  of  candidates,  eliminate  those  that  are  not  likely  to 
be  effective  (e.g.,  those  not  familiar  to  the  learners,  those  difficult  to  understand,  and/or 
those  that  do  not  closely  match  the  relevant  attributes  of  the  subject).  Choose  the  analog  or 
analogs  from  those  that  remain. 

•  D:  Describe  the  connector  and  ground.  Provide  a  clear  and  thorough  description  of  the 
similarities  and  differences  between  the  subject  and  analog. 

When  presenting  an  analogy,  they  recommend  follow  the  memory  aid  ADAPT: 

•  A:  Analogize  early.  Present  the  analogy  during  the  initial  description  of  the  subject. 

•  D:  Deliver  in  combination  format.  Use  combinations  of  verbal  and  pictorial  formats  (or 
other  media  combinations). 

•  A:  Assimilate  elaborations.  As  you  elaborate  the  subject,  assimilate  the  added 
information  to  the  analog. 

•  P:  Provide  practice.  Allow  the  learners  to  apply  the  subject  information  and  provide 
corrective  feedback. 

•  T:  Test.  Provide  new  situations  where  the  learner  can  apply  the  subject  information. 
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2.b.  The  Learner  is  Required  to  Interact  with  Content 

“Requiring”  interaction  means  that  the  course  should  be  designed  so  that  the  learner  must 
engage  in  some  form  of  activity  where  he  or  she  uses  the  content  in  some  meaningful  way. 

Using  the  Scale 

This  scale  is  used  to  evaluate  the  course  in  terms  of  how  it  is  designed  to  promote  interaction 
between  the  learner  and  the  content.  Use  the  anchor  statements  below  the  scale  to  help  assign  a 
rating  on  this  criterion.  Additional  information  on  this  criterion  is  provided  in  the  discussion 
below. 

Terms.  Concepts,  Techniques,  and  Supporting  Literature 

All  of  the  activities  discussed  under  Criterion  2. a  (relevance)  are  also  applicable  here. 
However,  in  addition  to  relevance,  the  activities  should  be  designed  to  help  the  learner  interact 
with  the  material  to  be  learned.  Interactive  activities  help  the  learner  to  integrate  new 
information  into  existing  mental  structures  and/or  develop  the  necessary  motor  skills  required  to 
perform  the  task.  In  all  cases,  the  learner  should  be  assessed  (formally  or  informally)  on  how  his 
or  her  performance  demonstrates  that  he  or  she  is  meeting  the  instructional  objectives  (see 
Evaluation  Area  3  in  Chapter  5)  and  should  be  provided  feedback  on  the  quality  of  his  or  her 
performance  (see  Evaluation  Area  4  in  Chapter  6). 

Najjar  (1988)  defines  interaction  as  “mutual  action  among  the  learner,  the  learning  system, 
and  the  learning  material”  (p.  314).  He  cites  several  studies  that  show  a  positive  effect  on 
learning  with  computers  when  learners  use  an  interactive  interface.  However,  this  interactive 
interface  is  only  beneficial  if  it  encourages  learners  to  cognitively  process  the  learning  material. 
In  other  words  the  material  must  be  cognitively  engaging  (see  Criterion  2.c). 

Types  of  Interactions.  Sims,  Dobbs,  and  Hand  (2001)  discuss  several  types  of  interactivity 
that  encompass  both  human-computer  interactions  and  human-human  interactions.  These 
include  interactions  between:  learner  and  learner,  learner  and  teacher,  teacher  and  content, 
teacher  and  teacher,  learner  and  interface,  and  learner  and  content.  It  is  only  the  last  two  (learner 
and  interface  and  learner  and  content)  that  are  within  the  scope  of  this  guide  since  the  focus  is  on 
the  evaluation  of  asynchronous  instruction  without  the  presence  of  an  instructor  or  other 
collaborative  learners.  Learner  and  interface  interactions  deal  with  how  the  learner  navigates 
through  the  course  and  provides  various  inputs  when  required.  Learner  and  interface  criteria  are 
evaluated  in  the  User  Interface  section.  Learner  and  content  interactions  are  evaluated  under  this 
criterion. 

Interaction  with  Content.  One  definition  of  learning  is  “a  relatively  permanent  change  in 
someone’s  knowledge  based  on  the  person’s  experience”  (Meyer,  2002,  p.  3).  Once  the  person 
has  learned  the  new  information,  he  or  she  needs  to  be  able  to  use  it  to  help  learn  new  things  or 
to  perform  a  task.  This  is  usually  referred  to  as  transfer  of  learning.  Transfer  is  facilitated  by 
meaningful  instruction  and  interaction  with  content  (Meyer,  2002).  Some  types  of  cognitive 
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interaction  with  information  include  thinking  about  the  information,  reflecting  on  its 
implications,  discussing  how  it  relates  to  other  topics  in  the  course,  hypothesizing  how  the  new 
information  could  improve  task  performance,  comparing  one  source  of  information  to  another, 
and  classifying  the  information  into  job-relevant  categories.  These  types  of  interactions  promote 
active  learning,  which  has  been  shown  to  be  more  effective  than  passive  learning  (e.g., 
Dickinson,  1978).  Meyer  (2002)  discusses  numerous  instructional  methods  to  help  learners 
interact  with  content  to  promote  meaningful  learning.  These  include:  1)  presentation  of  concrete 
examples,  2)  methods  to  make  examples  more  relevant  (e.g.,  case-based  learning),  3)  guiding 
cognitive  processes,  4)  teaching  strategies  to  make  learning  active,  and  5)  teaching  problem¬ 
solving  strategies. 

Another  activity  that  promotes  learning  is  to  have  the  learners  produce  information  rather 
than  being  presented  with  the  information.  Called  the  “generation  effect,”  (Slamecka  &  Graf, 
1978)  it  “is  one  of  the  most  robust  findings  in  the  experimental  psychology  literature” 
(deWinstanley  &  Bjork,  2002,  p.  22).  These  authors  indicated  that  the  generation  effect  has  been 
shown  in  a  variety  of  learning  areas  from  mathematics  to  reading  comprehension. 

Practice.  “One  major  activity  separates  the  gifted  amateur  from  the  professional:  practice” 
(Norman,  1976,  p.  199).  Many  studies  have  shown  the  advantages  of  spreading  practice  sessions 
over  time  (distributed  practice)  as  compared  to  a  single  practice  session  (massed  practice).  For 
example,  Duncan  (1951)  allowed  one  group  to  practice  in  one  session  that  was  three  times  as 
long  as  the  total  time  another  group  practiced  in  distributed  sessions.  He  found  that  the 
distributed  practice  group  outperformed  the  massed  practice  group. 

How  does  one  tell  if  the  learner  has  practiced  enough?  Several  sources  of  data  indicate  that 
more  practice  is  always  better.  Experts  at  difficult  tasks  (e.g.,  playing  the  piano  or  flying  a  high- 
performance  jet)  continuously  practice  to  refine  their  skills  and  keep  them  current  (Schneider, 
1985).  It  has  also  been  shown  that  the  overlearning  of  a  task  can  result  in  better  retention  of 
skills.  Overlearning  occurs  when  learners  engage  in  additional  practice  after  they  have  learned  a 
task  to  an  established  criterion  level.  The  additional  practice  helps  them  “automate”  the  learned 
skills  and  helps  ensure  that  they  will  be  able  to  perform  the  skill  after  a  prolonged  retention 
interval  or  under  stressful  conditions. 

Schendel  and  Hagman  (1982)  investigated  the  effects  of  overlearning  for  a 
disassembly/assembly  task.  They  trained  three  groups  of  soldiers  until  they  could  all  perform 
the  task  to  criterion  levels.  The  control  group  received  no  additional  training.  The  experimental 
group  received  100%  overtraining  on  the  task.  Eight  weeks  later,  the  third  group  received 
refresher  training  just  before  they  rejoined  the  other  groups  for  a  retention  test.  The  first  trial  on 
the  retention  test  served  as  the  measure  of  retention.  The  results  showed  that  both  the 
overlearning  and  retention  groups  outperformed  the  control  group.  The  performance  of  the 
retention  and  overlearning  groups  did  not  differ.  The  results  indicate  that  overlearning  is  as 
effective  improving  retention  as  retraining  on  the  tasks. 

Hagman  and  Rose  (1983)  reviewed  13  experiments  on  the  retention  of  learned  tasks  in  the 
military  context.  Some  of  their  conclusions  from  this  review  are  as  follows: 
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•  Retention  can  be  improved  with  repetition  (practice)  both  before  and  after  task 
proficiency  has  been  achieved.  This  supports  the  overlearning  principle. 

•  Retention  is  better  when  task  repetitions  are  spaced  rather  massed  during  training. 

•  Retention  can  be  improved  through  the  use  of  instructional  methods  that  are  tailored 
for  the  specific  training  environment.  For  example,  if  training  is  to  be  conducted  in  a 
military  field  environment,  the  specific  variables  that  make  this  environment  unique 
need  to  be  determined  and  accounted  for  when  designing  the  instructional  approach. 


The  Institute  for  Higher  Education  Policy  (IHEP)  in  their  evaluation  framework  for  e-leaming 
maintains  that  courses  should  be  designed  to  require  learners  to  engage  in  analysis,  synthesis, 
and  evaluation  as  part  of  course  requirements  (Broadbent  and  Cotter,  2002).  The  Texas 
Education  Agency  (2003)  recommends  that  instructional  interactions  should  vary  according  to 
the  desired  type  of  learning  outcomes.  For  example,  learners  who  need  to  learn  about  concepts 
might  engage  in  an  activity  where  they  have  to  identity  and  categorize  the  differences  and 
similarities  among  concepts.  Learners  required  to  learn  a  procedure,  might  participate  in  a 
simulated  activity  where  they  have  to  arrange  procedures  sequentially  or  identify  a  procedural 
step  that  is  incorrect.  Learners  learning  psychomotor  skills  might  need  to  interact  with  a  hands- 
on  simulation  or  an  equipment  mock-up. 

The  content  should  be  presented  in  such  a  way  that  it  encourages  interaction  by  asking  for 
varied  kinds  of  inputs  (Bangert-Drowns  and  Kozma,  1989).  However,  there  should  not  be  too 
many  inputs  required  and  they  should  be  relevant  to  the  instructional  objectives,  not  “just  for  the 
sake  of  it.” 

Thibodeau  (1997),  an  instructional  developer  and  designer  of  interactive  courseware  for  the 
U.  S.  Army,  provides  the  following  two  suggestions  for  designing  content  to  encourage  learner- 
content  interaction. 

•  Break  the  content  into  small  units,  each  followed  by  an  interactive  exercise 

•  The  interactions  (e.g.,  questions)  should  provide  an  opportunity  for  the  learner  to  use 
what  they  have  just  learned. 

Two  techniques  that  can  be  used  to  increase  the  level  of  learner-content  interaction  are  to 
enable  learners  to  annotate  text  or  summarize  the  information  in  their  own  words.  Some 
research  indicates  that  “annotating  text  while  reading  improves  comprehension”  (Dembo  and 
Junge,  2003,  p.  58).  There  is  additional  research  that  indicates  that  learners  “who  summarize 
readings  comprehend  and  recall  more  than  those  who  do  not”  (Dembo  and  Junge,  2003,  p.56). 
Course  designers  could  include  mechanisms  that  allow  learners  to  engage  in  these  interactive 
activities. 

Although  most  research  on  interaction  has  been  conducted  in  the  context  of  lecture-based  or 
web-supplemented  courses  (using  the  web  as  an  adjunct  to  the  lecture,  such  as  posting  the 
syllabus,  answering  questions,  or  conducting  discussion  groups),  it  is  reasonable  to  assume  that 
interaction  is  also  an  important  component  of  learning  in  asynchronous  courses.  Milheim  (1995) 
believes  that  “the  interaction  between  the  computer  and  the  learner  may  be  one  of  the  most 
important  aspects  of  effective  computer-based  learning”  (p.  163).  Zirkin  and  Sumler  (1995) 
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concluded  from  their  review  of  the  research  literature  that  “The  weight  of  evidence. .  .was  that 
increased  learner  involvement  by  immediate  interaction  resulted  in  increased  learning...” 

(quoted  in  Roblyer  and  Ekhaml,  2000,  p.  1).  This  is  also  true  for  computer-  or  web-delivered 
instruction.  However,  in  the  authors’  experience,  modeling  the  role  of  the  instructor  is  one  of  the 
most  important,  yet  difficult  aspects  in  designing  computer-  or  web-delivered  instruction. 
Instructional  developers  need  to  try  and  put  themselves  into  the  role  of  the  learner — anticipating 
frequently  asked  questions  and  areas  of  confusion.  Content  and  activities  should  address  these 
questions  and  areas  because  there  is  no  instructor  to  fill  this  role.  Formative  evaluations  are  a 
good  method  to  help  identify  these  important  issues  and  questions. 

In  any  case,  it  is  more  likely  that  learners  will  interact  with  content  if  it  is  presented  in  an 
engaging  manner.  Evaluating  how  much  of  the  content  engages  the  learner  is  the  topic  of  the 
next  criterion. 

Note:  Roblyer  and  Ekhaml  (2000)  and  Roblyer  and  Wiencke  (2003)  provide  a  rubric  for  the 
design  of  interactivity  for  instructor-mediated  distance  learning  courses.  For  those  needing  to 
evaluate  instructor-mediated  courses,  an  adaptation  of  their  rubric  is  provided  in  Appendix  D. 


2.c.  Instruction  is  Engaging  (Attracts  and  Maintains  Learners’  Attention) 

In  order  to  leam,  learners  must  attend  to  the  instruction.  One  of  the  most  important  design 
decisions  is  how  to  gain  and  maintain  the  learners’  attention,  or  how  to  “engage”  them  with  the 
content. 

Using  the  Scale 

This  scale  is  used  to  evaluate  the  course  in  terms  of  how  much  of  the  content  engages  the 
learner  (attracts  and  maintains  attention).  Use  the  anchor  statements  below  the  scale  and  the  list 
of  examples  to  help  assign  a  rating  on  this  criterion.  Additional  information  to  help  understand 
this  criterion  can  be  found  in  the  discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

Issues  of  engagement  are  closely  related  to  those  of  motivation  (see  discussion  of  ARCS 
Model  under  Criterion  1  .a).  Richard  Clark  (2003)  maintains  that  course  designers  can  increase 
and  maintain  engagement  by  “helping  students  connect  their  personal  goals  and  interests  to 
course  goals, . . .  and  by  helping  students  maintain  their  confidence  in  achieving  the  course 
goals”  (p.  20).  He  also  believes  that  there  are  three  types  of  motivational  goals  for  instruction: 

1)  active  engagement  (motivating  the  learner  to  start  the  learning  activity);  2)  persistence 
(motivating  the  learner  to  continue  to  work  towards  the  instructional  goal  in  a  focused  way, 
despite  distractions);  and  3)  mental  effort.  In  attempting  to  help  learners  invest  mental  effort, 
Clark  cautions  that  “instructional  strategies  and  complex  screen  displays  risk  overloading 
working  memory  and  causing  ‘automated’  cognitive  defaults  where  mental  effort  is  both  reduced 
and  directed  to  nonleaming  goals”  (p.  23). 
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Nash  (2003,  p.  5)  observes  that  there  are  always  multiple  ways  to  achieve  an  instructional 
objective.  Even  the  most  interesting  and  well-designed  presentation  format  can  become  boring  if 
it  is  overused.  It  is  more  likely  that  the  learners’  attention  will  be  maintained  if  a  variety  of 
methods  are  used  for  presentations  and  demonstrations. 

Some  of  these  methods  include  audio  and  video  lectures,  audio  and  video  testimonials  by 
experts,  animations,  videos  of  real-life  examples,  or  even  still  graphics  combined  with  text  (as 
well  as  the  methods  and  activities  discussed  under  Criterion  2.a).  Whatever  combination  of 
methods  is  used,  developers  should  ensure  that  each  method  is  carefully  designed  to  use  its 
strengths  and  avoid  its  weaknesses.  Several  other  techniques  have  been  recommended  to 
enhance  learner  motivation  and  engagement  with  instructional  content  (see  Allessi  &  Trollip, 
1991;  deWinstanley  &  Bjork,  2002). 

Attention.  The  content  should  be  designed  to  gain  and  maintain  attention  throughout  each 
lesson.  Techniques  to  provoke  attention  can  be  divided  into  two  groups: 

•  Those  that  are  involved  with  the  appearance  of  the  content  (e.g.,  the  Use  of  color, 
brightness,  graphics,  and  animation).  Care  should  be  taken  to  ensure  that  the 
combination  of  these  techniques  result  in  aesthetically  pleasing  pages  (see  section  6.c). 

•  Those  involved  with  the  presentation  and  organization  of  the  content  (e.g.,  the  use  of 
sound,  music,  humor,  novelty,  or  storytelling). 

Gaining  Attention.  Several  methods  have  been  suggested  to  gain  the  attention  of  learners, 
such  as  using  the  novelty  and  complexity  of  the  stimulus.  Novelty  of  a  stimulus  gains  attention 
because  it  arouses  the  curiosity  of  the  learner  to  direct  him  or  her  to  the  stimulus.  Travers 
suggests  three  ways  to  make  a  situation  novel:  1)  using  a  stimulus  that  has  not  been  presented 
recently;  2)  presenting  the  stimulus  in  a  new  or  unfamiliar  context  or  setting;  and  3)  presenting  a 
stimulus  that  has  never  been  used  before. 

Travers  (1972)  cites  research  that  suggests  individuals  are  more  likely  to  attend  visually  to  an 
object  that  is  more  complex  than  to  one  which  is  less  complex.  For  example,  babies  will  fixate 
on  a  more  complex  diagram  hanging  over  their  heads  than  a  simple  one.  People  also  tend  to 
visually  attend  more  to  a  person’s  eyes  and  mouth  rather  than  any  other  portions  of  the  face. 

This  may  be  because  there  is  more  information  in  a  smaller  space. 

Care  must  be  taken  when  using  attention-getting  techniques.  For  example,  humor  that 
illustrates  a  point  or  helps  place  it  in  the  context  of  a  real-world  situation  is  useful.  Humor  that 
is  not  used  appropriately  can  actually  become  a  distraction.  Thus,  it  is  important  to  creatively 
mix  the  attention-gaining  techniques,  but  only  to  support  well  organized  content  that  helps  the 
learner  integrate  concepts  that  are  built  upon  one  another  (see  Criterion  1  .a).  To  accomplish  this, 
the  techniques  and  interactive  activities  should  always  be  situated  in  meaningful,  job-relevant 
contexts  (see  Criterion  l.f). 

Maintaining  Attention.  It  is  not  sufficient  to  only  gain  attention,  the  learner’s  attention  must 
be  directed  to  the  information  to  be  learned  and  it  must  be  maintained  until  the  information  is 
learned.  Patricia  deWinstanley  and  Robert  Bjork  (2002)  point  out  that  we  have  known  for 
decades  that  “divided  attention  results  in  poorer  memory  than  full  attention  does”  (p.  20).  They 
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further  observe  that  recent  research  indicates  that  divided  attention  is  most  detrimental  during  the 
encoding  of  information  (Iidaka,  Anderson,  Kapur,  Cabeza,  &  Craik,  2000;  Naveh-Benjamin, 
Craik,  Gravrilescu,  &  Anderson,  2000). 

The  term  vigilance  has  come  to  be  used  as  synonymous  with  maintaining  attention.  Most 
vigilance  research  has  followed  similar  procedures.  Subjects  are  asked  to  monitor  something 
like  a  radar  screen  until  a  designated  signal  appears.  The  subject’s  task  is  to  report  when  the 
signal  is  noticed.  One  of  the  earliest  vigilance  studies  was  conducted  during  World  War  II  by  N. 
H.  Mackworth,  a  British  psychologist.  Mackworth’s  results  demonstrated  that  during  a  two-hour 
period,  “the  subject’s  ability  to  detect  a  specified  signal  markedly  decreased  each  half  hour” 
(Magill,  1980,  p.  103). 

Other  vigilance  studies  have  focused  on  the  factors  that  affect  a  person’s  ability  to  maintain 
attention.  For  example,  Wilkinson  (1963)  examined  the  effects  of  sleep  deprivation  on  subjects’ 
ability  to  tap  a  series  of  specified  points  when  corresponding  lights  appeared  in  a  certain  order. 
The  results  indicated  that  most  errors  were  committed  by  subjects  who  were  sleep  deprived  and 
who  performed  the  task  in  a  quiet  situation.  The  least  errors  were  committed  by  subjects  with 
normal  sleep  who  performed  in  a  quiet  situation.  With  sleep  deprived  subjects,  a  noisy 
environment  seemed  to  help  performance,  whereas  the  performance  of  subjects  with  normal 
sleep  was  hindered  by  noise. 

Broadbent  (1958,  chapter  6)  reviewed  what  he  termed  “theories  of  vigilance  decrement.” 

One  theory  postulates  that  people  lose  attention  because  the  surroundings  are  monotonous. 
Another  suggests  that  if  a  signal  is  very  infrequent,  an  individual’s  arousal  level  will  deteriorate 
during  the  time  of  the  activity.  A  third  theory  suggests  that  the  incoming  information  is 
“filtered”  in  some  way  so  that  only  part  of  the  information  reaches  the  perceptual  system.  Each 
of  these  theories  may  contribute  to  the  explanation  of  vigilance  problems. 

These  theories  and  vigilance  research  data  suggest  several  ways  instructors  and  instructional 
developers  can  aid  the  maintenance  of  attention.  First,  instruction  can  sensitize  learners  to 
important  cues  that  help  the  learner  prepare  to  detect  a  signal.  For  example,  a  baseball  coach  can 
instruct  an  outfielder  to  make  each  pitch  a  signal  event  to  begin  preparation  that  directs  the 
player’s  attention  to  a  hit  ball.  Words  of  encouragement  or  instructions  directed  toward  the 
learner  can  also  help  prevent  his  or  her  attention  from  wandering.  Thirdly,  Mackworth  found 
that  providing  immediate  knowledge  of  results  after  each  of  the  signals  to  be  detected  maintained 
subjects’  proficiency  for  longer  periods  of  time.  Finally,  the  instructional  environment  should 
never  be  permitted  to  become  boring  or  monotonous.  Practice  drills  should  be  kept  short  enough 
that  they  remain  effective. 

Challenge.  A  learner  is  more  likely  to  remain  engaged  with  content  if  it  is  designed  to 
provide  and  maintain  increasing  levels  of  challenge.  Csikszentmihalyi  (1990)  has  examined  how 
persons  perform  optimally  in  a  variety  of  situations,  from  sports  to  academics.  He  uses  the  term 
flow  to  describe  “the  state  in  which  people  are  so  involved  in  an  activity  that  nothing  else  seems 
to  matter;  the  experience  itself  is  so  enjoyable  that  people  will  do  it  even  at  great  cost,  for  the 
sheer  sake  of  doing  it”  (p.4).  The  flow  experience  occurs  when  we  find  the  balance  between 
challenge  and  boredom.  As  one  develops  greater  knowledge  or  skills,  learning  activities  must 


96 


Technical  Report  2005-002 


offer  more  challenge  to  keep  the  learner  from  becoming  bored.  However,  if  the  learning  activity 
is  too  challenging,  the  learner  may  become  anxious  and  withdraw  his  or  her  attention.  Thus, 
facts,  concepts,  and  activities  should  increase  in  complexity  and  difficulty  as  the  learner 
progresses  through  the  course,  but  at  a  reasonable  rate.  Developers  can  determine  a  reasonable 
rate  of  increasing  challenge  by  conducting  formative  evaluations  of  the  content  with  a  small 
number  of  learners. 

Competition  can  be  used  as  a  technique  to  help  keep  the  learner  engaged  as  he  or  she 
progresses  toward  greater  levels  of  difficulty.  There  are  at  least  four  types  of  competition  that 
can  be  used  in  an  instructional  product:  competition  with  other  learners,  with  the  computer,  with 
the  clock,  or  with  him-  or  herself.  It  is  important  that  the  learner  understands  the  goals  of  the 
competition  and  that  these  goals  are  stated  in  an  unambiguous  manner  (e.g.,  the  number  of 
questions  that  must  be  answered  correctly  in  order  to  advance  to  the  next  level  or  section). 

A  difficult  task  for  the  evaluators  using  this  tool  is  to  determine  whether  the  content 
“challenges”  the  learner.  How  can  someone  looking  at  a  course  determine  if  it  is  challenging? 
There  is  no  definitive  answer  to  this  question.  However,  the  evaluator  can  assume  a  higher  level 
of  challenge  if  the  structure  of  the  course  includes  some  of  the  following  characteristics: 

•  Learning  objectives  are  based  on  critical  and  difficult  job  tasks  (Some  evaluators  [e.g., 
program  managers]  may  not  have  access  to  this  information,  but  course  developers 
should  keep  this  in  mind  throughout  design  and  development.) 

•  Content  progresses  from  easy  to  difficult  information 

•  Content  progresses  from  simple  to  complex  activities 

As  mentioned  previously,  another  way  to  challenge  the  learner  is  through  activities  that 
encourage  him  or  her  to  generate  information.  This  activity,  sometimes  called  the  generation 
effect,  has  been  shown  to  be  a  powerful  method  of  learning  (Slamecka  &  Graf,  1978; 
deWinstanley  &  Bjork,  2002). 

Confidence.  Another  technique  to  help  keep  the  learner  engaged  is  to  maintain  his  or  her 
confidence  that  they  are  meeting  the  instructional  objectives  of  the  course.  That  means  the 
objectives  must  be  clearly  stated  (l.b)  and  the  instructional  content  designed  to  support  them 
(l.c).  In  addition,  the  learner  should  be  given  opportunities  to  be  successful  as  he  or  she  engages 
in  learning  activities  and  given  encouragement  (reinforcement)  when  performing  as  expected 
(4.c).  Engagement  can  be  enhanced  by  using  adjunct  reinforcement,  such  as  following  a  less 
engaging  activity  (e.g.,  a  drill)  with  an  enjoyable  activity  (e.g.,  a  game). 

Stuart,  Brown,  and  Draper  (2004)  allowed  learners  to  use  an  electronic  voting  machine 
(handset)  to  answer  questions  and  obtain  feedback  in  class.  The  researchers  hypothesized  that 
the  learners  would  ask  more  questions  and  be  more  open  to  feedback  because  the  system  allowed 
them  to  retain  their  anonymity.  They  found  that  the  learners  liked  using  the  handsets  and  felt 
more  confident  in  their  learning.  The  effectiveness  of  this  approach  has  not  yet  been  compared 
to  traditional  classroom  lectures  or  other  instructional  approaches.  However,  in  computer-  and 
web-delivered  courses,  without  an  instructor’s  presence,  the  implication  is  that  non-graded 
assessments  where  learners  can  take  risks  and  receive  feedback  might  build  their  confidence  and 
increase  their  level  of  engagement. 
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Lack  of  confidence  in  their  ability  to  interact  with  the  internet  has  been  shown  to  be 
associated  with  opposition  to  web-based  instruction  (Thompson  &  Lynch,  2003).  These 
researchers  surveyed  257  college  students  and  found  that  those  with  weak  Internet  self-efficacy 
beliefs  were  also  inclined  to  resist  web-based  instruction  (r  =  -.29).  The  design  of  the  user 
interface  is  one  way  to  help  the  learner  develop  and  maintain  self  confidence.  Discussions  on 
how  to  design  learner-friendly  interfaces  are  found  in  the  chapters  in  Section  III. 

Maintaining  satisfaction.  If  a  learner  is  satisfied  that  he  or  she  is  attaining  the  objectives  of 
the  course,  it  is  more  likely  that  he  or  she  will  remain  engaged.  A  learner’s  satisfaction  can  be 
maintained  by  giving  supportive  feedback  (see  section  4)  that  encourages  the  learner  and  keeps 
the  learner  informed  of  how  well  he  or  she  is  progressing  through  the  course.  The  feedback  is 
likely  to  be  more  effective  if  it  shows  learners  how  what  they  are  learning  can  be  useful  to  them. 

Curiosity  and  fantasy.  A  learner’s  inherent  curiosity  can  be  used  as  a  tool  to  maintain 
engagement.  Sensory  curiosity  can  be  aroused  by  the  use  of  novel,  unexpected  sounds  and 
visual  stimuli.  Cognitive  curiosity  can  be  aroused  by  incorporating  elements  of  surprise  or  by 
giving  only  partial  information  and  then  stimulating  the  learner’s  desire  to  know  more.  A 
learner’s  natural  tendency  to  fantasize  can  also  be  used  to  keep  them  engaged.  For  example, 
engagement  can  be  improved  by  having  the  learners  participate  in  an  imaginary  role-playing 
situation  that  helps  them  use  the  information  they  are  trying  to  learn. 

Curiosity  that  leads  to  exploratory  behavior  is  not  always  beneficial  for  learning  specific 
tasks.  Martens,  Gulikers,  and  Bastianes  (2004)  compared  learners  with  high  intrinsic  motivation 
to  those  with  lower  levels  of  intrinsic  motivation.  They  found  that  the  highly  motivated  learners 
engaged  in  more  explorative  study  behavior.  However,  the  learning  outcomes  of  these  learners 
were  not  better  than  the  learners  who  did  not  engage  in  as  much  explorative  study.  These  results 
indicate  that  explorative  behaviors  need  to  be  directed  to  ensure  that  they  focus  on  important 
instructional  information. 

Variety.  Learners  will  be  less  likely  to  become  bored  if  the  content  includes  a  variety  of 
presentation  methods  (e.g.,  text,  graphics,  color,  sound,  videos,  or  expert  testimonials).  Variety 
can  also  be  introduced  by  using  several  different  voices  (male  and  female)  for  narrations.  This 
also  makes  it  easier  to  update  a  course  because  the  original  narrator  does  not  have  to  be  located 
to  change  or  add  narrations. 

A  variety  of  response  options  (e.g.,  mouse,  keyboard,  touch-screens,  joysticks,  etc.)  can  also 
encourage  engagement.  However,  as  Bangert-Drowns  and  Kozma  (1989)  caution,  there  should 
not  be  too  many  inputs  required,  nor  should  required  inputs  change  “just  for  the  sake  of  it.” 
Furthermore,  the  learner  should  always  understand  which  type  of  input  is  required. 

Session  length.  Finally,  learners  should  not  be  expected  to  concentrate  for  inordinate  periods 
of  time.  Lessons  and  activities  should  be  short  enough  that  they  can  be  completed  before 
learners  become  bored.  Optimal  session  length  will  depend  on  task  type  and  complexity. 
Formative  evaluations  can  help  determine  the  maximum  length  of  each  instructional  segment. 
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2.d.  Instructional  Media  Directly  Support  Learning  Activities 

Instructional  media  can  be  very  attractive  and  can  provide  instructional  benefits.  However, 
they  can  also  distract  from  the  instruction  if  they  are  not  used  to  directly  support  learning 
activities. 

Using  the  Scale 

This  scale  is  used  to  evaluate  the  course  in  terms  of  how  many  instructional  media  found  in 
the  course  directly  support  learning  activities.  Use  the  anchor  statements  below  the  scale  to  help 
assign  a  rating  on  this  criterion.  Additional  information  to  help  the  evaluator  understand  this 
criterion  is  provided  in  the  discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

Every  effort  should  be  used  to  maintain  the  learners’  engagement  with  the  instructional 
content.  Media  (graphics,  videos,  animations,  etc.)  can  be  powerful  tools  to  engage  learners. 
However,  as  Ruth  Clark,  referring  to  the  work  of  Richard  Clark,  stated  “more  than  50  years  of 
research  indicate  that  what  causes  learning  is  not  the  technological  delivery  device  but  the 
instructional  methods  that  facilitate  human  cognitive  learning  processes”  (Clark,  R.  C.,  2001, 
P-29). 

Media  should  only  be  used  to  “facilitate  and  enhance  learning”  (Texas  Education  Agency; 
2003,  F.l)  by  directly  supporting  the  content  and  instructional  objectives.  Malcolm  (2000) 
maintains  that  many  courses  include  too  much  “glitz,”  which  he  defines  as  “gratuitous  graphics 
and  audio  that  do  not  support  learning,  much  less  doing”  (p.  2).  Media  should  never  be  included 
gratuitously  because  they  can  become  distractions.  When  an  instructional  medium  enhances  the 
instruction  and  engagement  it  should  be  used.  If  it  does  not  add  instructional  value,  it  should  be 
omitted.  Some  of  the  legitimate  uses  of  media  include: 

•  Simplifying  content 

•  Reinforcing  content 

•  Helping  the  learner  to  make  connections  among  pieces  of  information 

•  Illustrating  learning  goals 

Although  some  studies  evaluate  the  effectiveness  of  whole  multimedia  applications  (e.g., 
Hansen,  Resnick,  &  Galea,  2003),  they  seldom  systematically  evaluate  the  specific  multimedia 
characteristics  that  lead  to  effectiveness.  The  work  of  Mayer  and  his  associates  is  a  notable 
exception  (Mayer,  2003;  Mayer  &  Anderson,  1991).  A  detailed  discussion  of  media  principles  is 
provided  in  Section  6.e. 
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Chapter  5: 

Performance  Assessment 
(Evaluation  Area  3) 

Learners  need  to  interact  with  instructional  content  in  order  to  incorporate  new  information 
into  their  existing  mental  structures  (knowledge)  and  to  develop  proficiency  on  the  actions  that 
support  the  tasks  being  trained  (skills).  It  is  essential  that  learners  be  assessed  to  determine  if 
they  are  learning  what  is  intended  and  at  the  required  level  of  proficiency.  However,  the 
American  Association  for  Higher  Education  (AAHE,  2004)  warns  that  assessments  must  be 
linked  to  specific  goals  (e.g.,  instructional  objectives;  see  criteria  l.a  and  l.b)  or  they  risk 
becoming  an  “exercise  in  measuring  what’s  easy,  rather  than  a  process  of  improving  what  we 
really  care  about”  (p.l).  Furthermore,  they  should  not  be  an  end  in  themselves,  but  rather  a 
“vehicle  for  educational  improvement”  (p.  1).  Assessments  can  be  formal  (e.g.,  results  of 
quizzes)  or  informal  (participation  in  a  no-graded  activity).  The  criteria  discussed  in  this  chapter 
concern  the  design  of  these  assessments.  The  following  chapter  discusses  criteria  on  how  to 
provide  feedback  to  the  learner  on  the  results  of  the  assessments. 


3.a.  Assessments  are  Relevant 


It  is  generally  agreed  that  the  performance  of  learners  needs  to  be  assessed  regularly  during 
instruction  to  ensure  that  they  are  on  track  and  learning  the  required  material.  These  assessments 
are  only  useful  if  they  meet  these  goals.  Any  assessment  that  does  not  measure  learner 
performance  on  knowledge  and  skills  that  support  the  objectives  of  the  course  should  not  be 
included. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  many  of  the  assessments  included  in  the  course  are 
relevant  to  the  needs  of  the  learners  and  the  instructional  objectives  of  the  course.  Use  the 
anchor  statements  and  list  of  characteristics  of  relevant  assessments  found  below  the  scale  to 
help  assign  a  rating  on  this  criterion.  If  you  assign  a  rating  of  one  (1)  on  this  criterion,  you  must 
also  assign  a  rating  of  one  (1)  on  Criteria  3.b  and  3.c.  Additional  information  on  this  criterion  is 
provided  in  the  discussion  below. 

Terms.  Concepts.  Techniques,  and  Supporting  Literature 


To  be  considered  relevant,  assessments  should  be  connected  to  issues  and  questions  that  the 
learner  really  cares  about  (AAHE,  2004).  This  means  that  each  and  every  assessment  should,  at 
a  minimum,  be  linked  to  the  instructional  objectives  of  the  course  and  should  assess  what  the 
objective  says  it  will  assess.  In  addition,  adult  learners  care  about  being  able  to  do  their  job 
more  effectively  and  advancing  in  their  careers.  Thus,  assessments  should  measure  the  skills  and 
knowledge  that  are  required  on  the  learner’s  job.  Furthermore,  they  should  measure  not  just 
what  the  learners  know,  but  what  they  can  do  with  what  they  know  (AAHE,  2004).  This  can  be 
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accomplished  by  designing  the  assessments  to  measure  more  than  just  factual  recall.  A  well- 
designed  assessment  will  require  the  learner  to  show  understanding  of  the  material  by  solving 
problems,  relating  pieces  of  information  together,  synthesizing  common  issues  or  identifying 
critical  differences  across  different  examples,  or  demonstrating  skills  and  creativity  in  a  realistic 
simulation.  These  types  of  assessments  should  then  be  used  as  the  basis  for  feedback  that  is  also 
relevant  (see  Criterion  4.b). 

Assessments  not  only  help  determine  if  the  information  has  been  learned.  They  can  also  have 
a  beneficial  role  in  the  learning  process.  Carrier  and  Pashler  (1992)  conducted  four  experiments 
to  determine  the  learning  benefits  of  retrieving  an  item  from  memory.  They  found  that  learners 
who  attempted  to  retrieve  target  items  from  memory  outperformed  learners  who  merely  studied 
the  items.  They  conclude  that  these  results  provide  support  for  the  hypothesis  “that  retrieving  an 
item  from  memory  has  beneficial  effects  for  later  retention  above  and  beyond  the  effects  due  to 
merely  studying  the  item”  (p.  639).  Thus,  relevant  assessments  should  be  considered  another 
learning  activity. 

Dempster  (1996)  cites  numerous  studies  that  support  the  general  learning  benefits  of  taking 
tests.  He  also  cites  several  studies  that  demonstrate  the  “test-spacing  effect.”  This  “refers  to  the 
fact  that  spaced  tests  are  more  effective  than  massed  tests”  (p.  323).  This  effect  is  probably 
similar  to  the  distributed-practice  effect  discussed  under  Criterion  2.b  (e.g.,  Duncan,  1951). 

The  type  of  assessment  can  influence  learners  in  terms  of  how  much  they  study,  the  study 
approach  they  take,  and  what  content  they  learn.  Scouller  (1998)  found  that  learners  prepare 
differently  for  multiple  choice  exams  and  essay  assignments.  Learners  were  more  likely  to 
perceive  multiple  choice  exams  as  assessing  factual  knowledge  and  essays  as  requiring  deeper 
understanding.  Thus,  they  were  more  likely  to  use  surface  learning  approaches  (recall  and 
reproduction)  when  preparing  for  multiple  choice  exams.  Conversely,  they  were  more  likely  to 
use  deep  learning  strategies  (development  of  meaning  and  understanding)  when  preparing  for 
essay  assignments.  These  results  indicate  that  multiple  choice  assessments  may  be  more  useful 
when  attempting  to  capture  the  learner’s  ability  to  recognize  correct  material,  but  essays  may  be 
more  useful  for  assessing  the  learner’s  deeper  level  of  understanding.  This  is  not  to  say  that  a 
well  constructed  multiple-choice  can  never  capture  a  learner’s  deeper  level  of  understanding. 
However,  these  types  of  multiple-choice  questions  are  very  difficult  to  compose. 


3.b.  Assessments  are  Logical 

The  positive  effects  of  assessments,  discussed  above,  are  not  likely  to  be  achieved  if  they  are 
not  logically  designed.  The  learner  must  understand  the  purpose  of  the  assessments  and  how 
they  support  the  instructional  purposes  of  the  course  or  course  segment. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  many  of  the  assessments  used  in  the  course  are  logical. 
Use  the  anchor  statements  below  the  scale  and  the  list  of  features  of  logical  assessments  to  help 
assign  a  rating  on  this  criterion.  Remember,  if  you  assigned  a  score  of  one  (1)  on  the  previous 
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Criterion  (3. a.),  you  must  also  assign  a  score  of  one  (1)  on  this  criterion.  Additional  information 
on  this  criterion  is  provided  in  the  discussion  below. 

Terms.  Concepts,  Techniques,  and  Supporting  Literature 

How  do  we  determine  that  an  assessment  is  “logical?”  This  does  not  refer  to  the  “science”  of 
logic,  which  deals  with  the  canons  and  criteria  of  inference  and  demonstrations  of  the  principles 
of  reason.  In  the  context  of  instructional  quality,  logical  refers  to  “using  reason  in  an  orderly 
cogent  fashion”  (Webster’s,  1977).  An  assessment  is  logical  if  it  is  cogently  designed  to  support 
the  instructional  objectives  of  a  lesson  or  course.  Below  are  some  issues  that  should  be 
considered  to  determine  if  assessments  are  cogently  designed. 

Assessments  as  Process.  Assessments  should  be  part  of  an  ongoing  process,  not  “episodic” 
(AAHE,  2004,  p.  1).  This  means  that  the  assessments  should  be  constructed  as  a  linked  series  of 
activities  over  time  that  provides  a  way  of  monitoring  learner  progress  towards  meeting  the 
instructional  objectives  of  the  course.  Assessments  that  are  designed  in  this  manner  provide 
information  that  is  cumulative  and  that  is  focused  on  the  explicitly  stated  purpose  of  the  lesson 
and  course.  This  information  can  be  used  not  only  to  improve  learner  performance,  but  also  to 
improve  the  assessments  themselves. 

The  authors  have  met  course  developers  who  advocate  the  use  of  “discovery  assessments.” 
These  are  “free  play”  exercises  where  the  learner  explores  or  discovers  important  information 
and  this  “discovery”  is  the  assessment.  There  is  little  or  no  data  on  the  effectiveness  of  this 
approach.  However,  if  used,  one  should  be  cautious  to  avoid  frustrating  the  learner  with  endless 
loops  where  he  or  she  must  search  for  the  correct  response. 

Managing  Learner  Expectations.  Learners  should  be  prompted  to  help  them  understand  what 
types  of  responses  are  expected  and  what  is  considered  successful  performance  (e.g.,  the 
standards  against  which  their  performance  will  be  judged).  The  prompts  should  reflect  the  user’s 
language  rather  than  computer  codes  or  numbers.  Responses  should  require  the  least  amount  of 
typing  and  should  be  “forgiving”  enough  to  cope  with  errors  of  format,  such  as  capitalization 
(Reeves,  1994  as  summarized  in  Botha,  2004). 

Learner  expectations  can  be  managed  by  helping  them  set  realistic  learning  goals.  Magill 
(1980)  provides  several  guidelines  for  setting  goals  that  are  relevant  to  this  Tool. 

1 .  Set  objective  goals.  Goals  should  be  stated  in  terms  of  a  number  or  some  other  objective 
form  rather  than  “do  your  best.” 

2.  Set  goals  that  are  meaningful.  The  goal  should  have  meaning  to  the  person.  For 
example,  asking  a  learner  to  hit  6  out  of  10  targets  is  not  as  meaningful  as  stating  that  6 
out  of  10  is  an  above  average  score. 

3.  Set  goals  that  are  obtainable.  Challenging,  yet  obtainable  goals  tend  to  increase 
motivation.  However,  an  unobtainable  goal  may  lead  to  failure  and  poorer  performance 
(Nelson,  1978). 

4.  Set  goals  on  the  basis  of  past  experience.  Learners  who  have  succeeded  in  the  past  will 
usually  expect  to  succeed  in  the  future.  Courses  should  be  designed  to  help  learners 
develop  a  pattern  of  success  (e.g.,  move  from  simple  to  complex  activities). 
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3.c.  Assessments  are  Varied 

Learners  can  become  bored  if  every  assessment  is  the  same  throughout  the  course.  To  make 
the  course  more  interesting,  relevant,  and  engaging,  a  variety  of  assessment  techniques  should  be 
used  at  different  places  in  the  course. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  many  different  types  of  assessments  are  used  in  the  course. 
The  anchor  statements  and  list  of  example  assessments  can  help  you  assign  a  score.  Remember, 
if  you  assigned  a  score  of  one  (1)  on  Criterion  3. a.  (relevance  of  assessments),  you  must  also 
assign  a  score  of  one  (1)  on  this  criterion.  Additional  information  to  help  understand  this 
criterion  is  provided  below. 

Terms.  Concepts,  Techniques,  and  Supporting  Literature 

The  American  Association  for  Higher  Education  recommends  that  assessments  be 
multidimensional  (e.g.,  employ  a  diverse  array  of  methods).  These  methods  should  include 
measures  of  knowledge  (e.g.,  quizzes)  and  also  measures  of  performance  (e.g.,  simulations,  role- 
play)  over  time  to  “reveal  change,  growth,  and  increasing  degrees  of  integration”  (AAHE,  2004, 
p.  1).  Variety  helps  maintain  the  learner’s  interest,  but  more  importantly,  it  provides  a  number  of 
diverse  perspectives  on  the  learner’s  performance. 
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Chapter  6: 

Performance  Feedback 
(Evaluation  Area  4) 


The  concept  of  feedback  originated  in  the  branch  of  engineering  known  as  control  theory 
(Moray,  1981).  A  special  case  of  feedback  (also  known  as  knowledge  of  results,  KOR)  has  long 
been  recognized  as  an  important  aspect  of  instruction.  Performance  feedback  is  the  information 
provided  to  the  learner  concerning  his  or  her  performance  on  the  assessments  (or  other 
interactive  activities)  in  an  instructional  event  (Annett,  1969;  Holding,  1965).  Performance 
feedback  can  be  used  as  a  reinforcing  stimulus  to  shape  and  maintain  behavior  (Fry,  1963; 
Skinner,  1968)  and  to  locate  and  correct  errors  (Anderson,  Kulhavy,  and  Andre,  1972;  Gagne 
and  Briggs,  1979;  Bardwell;  1981;  Holding,  1987).  It  has  even  been  shown  that  the  removal  of 
feedback  increases  a  person’s  annoyance,  boredom,  and  carelessness  (Elwell  and  Grindley, 
1938).  Based  on  years  of  research  on  the  value  of  KOR  for  a  large  variety  of  cognitive  and 
motor  tasks,  Magill  (1980)  states  that  it  is  considered  “as  the  strongest  and  most  important 
variable  involved  in  learning”  (p.  220).  Thus,  it  is  important  that  the  learner’s  performance  be 
assessed  throughout  the  instructional  process  and  that  he  or  she  receives  feedback  on  the 
adequacy  of  his  or  her  performance.  The  previous  chapter  included  discussions  on  criteria  to 
evaluate  the  design  of  assessments;  this  chapter  includes  discussions  on  criteria  to  evaluate  the 
design  of  performance  feedback. 


4.a.  Feedback  is  Timely 


Using  the  Scale 

This  scale  is  used  to  evaluate  the  timeliness  of  the  feedback  provided  to  the  learner.  Use  the 
anchor  statements  as  a  guide  to  help  assign  a  rating  on  this  criterion.  Remember,  if  you  assign  a 
score  of  one  (1)  on  this  criterion,  you  must  also  assign  a  score  of  one  (1)  on  Criteria  4.b,  4.c,  and 
4.d.  Additional  information  on  this  criterion  is  provided  in  the  discussion  below. 

Terms,  Concepts.  Techniques,  and  Supporting  Literature 

As  a  general  rule,  feedback  should  be  provided  immediately  or  soon  after  an  assessment. 
Learners  should  be  informed,  in  an  unambiguous  manner,  of  the  quality  of  their  performance. 
Then,  they  should  immediately  be  either  reinforced  for  correct  performance  or  provided 
remediation  for  incorrect  performance.  Learners  should  not  be  allowed  to  proceed  to  new 
information  that  requires  understanding  of  previous  information. 

In  a  classic  study,  Bilodeau,  Bilodeau,  and  Schumsky  (1959)  demonstrated  that  learners 
practicing  a  motor  task  without  feedback  performed  poorly.  However,  the  learners  who  received 
feedback  after  each  trial  improved  their  performance  consistently  over  twenty  trials.  When 
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feedback  was  withdrawn  after  either  two  or  six  trials,  the  learners’  performance  deteriorated  after 
that  point. 

Feedback  that  is  provided  too  soon  may  interfere  with  performance  of  the  task  itself.  For 
example,  for  a  simple  motor  task,  Weinberg,  Guy,  and  Tuppler  (1964)  found  that  the  optimal 
interval  for  feedback  was  5  seconds.  Feedback  given  1  second  after  task  completion  interfered 
with  task  performance.  Feedback  provided  after  10  and  20  seconds  did  not  result  in  either  better 
or  poorer  performance,  although  the  authors  speculate  that  with  longer  intervals,  some  forgetting 
may  occur.  Specific  timing  decisions  for  feedback  should  be  based  on  formative  testing  to 
ensure  optimal  feedback  intervals. 

One  should  also  be  cautious  about  providing  continuous  concurrent  feedback.  Concurrent 
feedback  is  “supplementary  information  presented  to  the  learner  during  the  actual  action” 
(Schmidt  &  Wulf,  1997,  p.  509).  Concurrent  feedback  (or  variants)  is  often  used  in  simulator 
training  (e.g.,  showing  pilots  their  location  with  respect  to  a  glide  path  during  landings). 
Performance  problems  may  occur  when  the  feedback  is  withdrawn  during  actual  task 
performance  because  they  can  become  dependant  on  the  feedback  as  a  performance  aid 
(sometime  called  a  “crutch”  effect).  “In  other  words,  continuous  concurrent  feedback  appears  to 
enhance  performance  during  practice  when  the  feedback  is  operating,  but  it  does  not  contribute 
to  learning  and  may  even  degrade  learning,  as  measured  on  retention  and  transfer  tests”  (Schmidt 
&  Wolf,  1997,  p.  510).  The  authors  point  out  that  there  are  a  few  exceptions  to  this 
generalization.  Most  notably,  very  complex  tasks  may  benefit  from  some  form  of  concurrent 
feedback.  However,  for  most  tasks,  it  is  probably  better  to  provide  feedback  shortly  after  task 
completion  or  allow  additional  practice  without  the  feedback  before  transfer  to  real-world 
performance. 


4.b.  Feedback  is  Meaningful 


Using  the  Scale 

This  scale  is  used  to  evaluate  how  much  of  the  feedback  in  the  course  is  meaningful  to  the 
learner.  Use  the  anchor  statements  below  the  scale  and  the  examples  of  meaningful  feedback  to 
help  assign  a  rating  on  this  criterion.  Remember,  if  you  assigned  a  score  of  one  (1)  on  Criterion 
4.a,  you  must  also  assign  a  one  (1)  on  this  criterion.  Additional  information  to  help  understand 
this  criterion  is  provided  below. 

Terms.  Concepts.  Techniques,  and  Supporting  Literature 

Although  feedback  is  widely  regarded  as  essential  to  quality  instruction,  it  is  not  always 
effective.  Kluger  and  DiNisi  (1998)  reviewed  a  large  amount  of  research  on  feedback  conducted 
in  both  natural  settings  and  laboratories.  They  reported  that  one  third  of  the  studies  showed  that 
feedback  had  a  negative  effect  on  performance.  Another  third  of  the  studies  showed  no  effect. 
“In  only  one  third  of  the  studies  did  feedback  increase  performance”  (Clark,  R.,  2003,  p.  18). 
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How  can  this  be?  Clark  believes  that  “learning  problems  appear  to  be  caused  by  feedback  that  is 
not  focused  on  clear  performance  goals  and  current  performance  results”  (p.  1 8). 

Practice  tests  can  be  useful  to  highlight  learning  goals  and  to  help  the  learner  focus  his  or  her 
attention.  Sly  (1999)  found  that  the  use  of  practice  tests  improved  the  performance  of  learners 
on  a  computer-managed  learning  package.  The  practice  tests  may  have  functioned  as  formative 
assessments,  providing  feedback  to  help  the  learners  concentrate  on  the  important  instructional 
information. 

Although  “it  is  doubtful  that  any  feedback  strategy  will  work  equally  well  for  all  learners” 
(Clark,  R.,  2003,  p.19),  feedback  is  more  likely  to  improve  learning  and  performance  if  it: 

•  Is  based  on  concrete  learning  goals  (instructional  objectives)  that  are  clearly  understood 
by  the  learners 

•  Describes  the  gap  between  the  learner’s  learning  goal  and  the  learner’s  current 
performance  and  suggests  how  to  close  the  gap 

•  Focuses  the  learner’s  attention  on  the  learning  goal  and  not  on  failure  to  achieve  the  goal 

Furthermore,  feedback  should  explain  or  show  how  what  the  learners  have  learned  can  be 
useful  to  them.  This  is  especially  true  for  adult  learners  who  need  to  transfer  their  learned 
knowledge  and  skills  to  the  work  place  (see  Criteria  l.f.  and  3. a). 

Bonk  (1999)  also  maintains  that  meaningful  feedback  should  include  a  combination  of  as 
many  of  the  following  as  possible.  It  should: 

•  provide  additional  information 

•  correct  errors 

•  ask  for  clarification 

•  debate  issues  and  explain  disagreements 

•  tie  concepts  to  one  another 

•  offer  insight  based  on  expert  experience 

In  addition,  feedback  should  be  objective;  not  based  on  the  preferences  of  a  particular  instructor 
or  expert. 


4.c.  Positive  Reinforcement  is  Provided  for  Correct  Responses 

Positive  reinforcement  increases  the  behaviors  that  it  follows.  It  also  serves  other  important 
instructional  functions,  as  discussed  below. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  often  positive  reinforcement  is  provided  as  part  of  the 
feedback  for  correct  responses.  Use  the  anchor  statements,  the  definition  of  positive 
reinforcement,  and  the  examples  to  help  assign  a  rating  on  this  criterion.  Remember,  if  you 
assigned  a  score  of  one  (1)  on  Criterion  4.a.  (timeliness  of  feedback),  you  must  also  assign  a 
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score  of  one  (1)  on  this  criterion  and  the  following  criterion  (4.d.).  Additional  information  on 
this  criterion  is  provided  in  the  discussion  below. 

Terms,  Concepts.  Techniques,  and  Supporting  Literature 

When  the  learner  has  performed  correctly  (e.g.,  provided  the  correct  answer  or  demonstrated 
a  skill  correctly)  the  feedback  should  acknowledge  and  reinforce  this  correct  performance  in  a 
positive,  encouraging  way.  There  is  sometimes  confusion  about  the  terms  reinforcement, 
reward,  and  punishment.  Reinforcement  always  strengthens  behaviors.  However  a  reinforcer 
can  be  either  positive  or  negative.  A  positive  reinforcer  is  any  event  (i.e.,  stimulus)  “which 
serves  to  strengthen  responses  preceding  its  occurrence”  (Baron,  Byrne,  &  Kantowitz,  1978,  p. 
120).  Positive  reinforcers  increase  the  probability  that  an  individual  will  respond  again  in  the 
way  he  or  she  responded  prior  to  experiencing  the  reinforcer.  This  is  what  most  people  think  of 
as  a  reward. 

A  negative  reinforcer  is  any  event  “which  strengthens  responses  leading  to  its  termination  or 
removal”  (Baron,  Byrne,  &  Kantowitz,  1978,  p.  120).  In  negative  reinforcement,  the  organism 
strengthens  behaviors  that  allow  it  to  avoid  or  terminate  uncomfortable  or  annoying  stimuli. 
When  a  negative  reinforcer  is  removed,  the  individual  is  more  likely  to  behave  the  way  he  or  she 
did  prior  to  the  removal  of  the  aversive  stimulus.  On  the  other  hand,  punishment  always  reduces 
behaviors.  A  punisher  is  some  type  of  unpleasant  stimulus  that  follows  a  behavior.  Its 
unpleasantness  reduces  the  likelihood  that  the  organism  will  behave  in  the  same  manner. 

Figure  1  illustrates  the  differences  between  positive  reinforcement,  negative  reinforcement 
and  punishment,  as  determined  through  experiments  with  rats.  The  top  row  of  the  figure 
illustrates  positive  reinforcement.  When  a  rat  presses  the  lever,  it  is  presented  with  food.  This 
increases  the  rat’s  tendency  to  press  the  lever.  On  the  other  hand,  the  second  row  of  the  figure 
illustrates  negative  reinforcement.  In  this  case,  the  rat  experiences  a  shock  until  it  presses  the 
lever.  This  behavior  turns  the  shock  off  and  results  in  an  increased  tendency  for  the  rat  to  press 
the  lever.  Both  types  of  reinforcement  increase  the  tendency  to  emit  a  behavior. 

Punishment,  as  illustrated  in  the  bottom  row  or  Figure  1  tends  to  decrease  behavior.  In  this 
example,  the  rat  experiences  a  shock  each  time  it  presses  the  lever.  This  results  in  a  decrease  in 
the  tendency  for  the  rat  to  press  the  lever. 

Years  of  research  with  animals  and  humans  support  these  general,  behavioral  principles.  In 
summary,  there  are  two  ways  to  increase  the  likelihood  of  a  behavior:  (1)  provide  a  reward 
(positive  reinforcement)  or  (2)  remove  an  aversive  stimulus  (negative  reinforcement).  In 
contrast,  there  are  also  two  ways  to  decrease  the  likelihood  of  a  behavior:  (1)  provide  an  aversive 
stimulus  (punishment)  or  (2)  take  away  the  rewarding  stimulus  that  follows  (is  contingent  on)  a 
response.  The  focus  on  this  criterion  is  on  positive  reinforcement,  geared  toward  increasing  the 
likelihood  of  a  correct  response  from  the  learner.  Punishment  is  never  recommended. 

Chi,  Siler,  Jeong,  Yamauchi,  and  Housmann  (2001)  reviewed  several  studies  on  the  use  of 
feedback.  They  concluded  that  feedback  can  “guide”  learners  in  the  sense  of  encouraging  them 
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to  either  stay  on  the  same  track  of  reasoning  or  problem  solving  (confirmatory  feedback)  or  to 
change  direction  or  goal  (remedial  feedback;  see  4.d  below). 


Behavior  Consequence 


Effect  on 
Behavior 


Positive 

Reinforcement 


Negative 

Reinforcement 


Response 
(press  lever) 


Aversive  Stimulus 
Removed 
(shock  turned  off) 


Tendency  to 
Press  Lever 
Increases 


Punishment 


Figure  1: 

Positive  and  Negative  Reinforcement  and  Punishment 


It  should  be  noted  that  positive  reinforcement  should  always  be  more  than  just  stating  that  the 
learner  has  given  a  “correct”  answer.  As  discussed  under  Criterion  4.b,  feedback  should  be 
meaningful.  Reinforcement  can  be  designed  to  be  another  learning  opportunity  by  helping  the 
learner  to  understand  the  relevance  of  his  or  her  correct  performance  and  how  this  performance 
can  be  used  as  the  foundation  for  building  associations  with  new  instructional  information. 

4.d.  Remediation  is  Provided  for  Incorrect  Responses 

Learners  do  not  always  perform  as  required  by  the  program  of  instruction.  When 
performance  is  incorrect,  the  course  should  not  just  tell  them  they  are  “wrong.”  Rather  it  should 
be  designed  to  provide  corrective  information  that  will  help  learners  understand  their  errors  and 
move  them  back  onto  the  correct  learning  path. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  often  remediation  is  provided  to  the  learner  when  he  or  she 
has  given  an  incorrect  response.  Use  the  anchor  statements  below  the  scale  and  the  list  of 
examples  of  remediation  to  help  you  assign  a  rating.  Remember,  if  you  assigned  a  score  of  one 
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(1)  on  Criterion  4.a,  you  must  also  assign  a  score  of  one  (1)  on  this  criterion.  Additional 
information  to  help  you  to  understand  this  criterion  is  provided  in  the  discussion  below. 

Terms.  Concepts.  Techniques,  and  Supporting  Literature 

When  a  learner  has  not  performed  as  desired  (e.g.,  provided  an  incorrect  answer  or  did  not 
demonstrate  correct  skill  performance),  Bangert- Drowns  and  Kozma  (1989)  recommend  that  the 
feedback  should: 

•  Provide  the  correct  answer  or  demonstrate  correct  skill  performance 

•  Explain  why  the  response  was  incorrect 

•  Encourage  the  learner  to  obtain  more  instruction  on  correct  behavior 

•  Lead  the  learner  to  this  remedial  instruction 

These  characteristics  of  feedback  make  it  more  meaningful  to  the  learner  (see  Criterion  4.b  for 
an  additional  discussion  on  meaningful  feedback). 

In  all  cases  the  feedback  should  be  provided  in  a  non-threatening,  encouraging  manner. 
Learners  can  be  demotivated  by  feedback  that  is  too  negative,  but  their  motivation  can  be 
increased  if  the  feedback  provides  the  tools  to  help  them  succeed  and  makes  them  feel  confident 
that  they  can  achieve  their  goals. 

Chi  (1996)  maintains  that  corrective  feedback  is  “critical  in  problem  solving  in  that  it 
effectively  reduces  searches  down  the  wrong  paths  of  a  search  space”  (p.  535).  The  course 
should  be  designed  not  only  to  provide  detailed  feedback  on  incorrect  responses,  but  also  to 
either  take  the  learner  back  through  the  discussion  or  to  state  the  instructional  information  in  a 
different  way  until  the  learner  has  mastered  the  content.  This  type  of  remediation  can  help 
learners  avoid  unproductive  “dead  ends”  and  incorrect  learning  paths. 
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Section  III: 

Detailed  Discussions  of 
User  Interface  Evaluation  Criteria 

The  following  discussions  provide  information  to  help  the  evaluator  understand  and  more 
confidently  assign  ratings  to  each  of  the  user  interface  evaluation  criteria.  The  discussions  begin 
with  directions  for  using  the  scales.  This  is  followed  by  definitions  of  terms  and  discussions  of 
concepts  and  techniques  to  maximize  the  quality  of  instructional  products  as  measured  by  each 
criterion.  The  discussions  also  provide  summaries  of  some  of  the  literature  that  supports  each 
criterion. 

Learner-Centric  Evaluations.  In  conducting  the  evaluation,  the  evaluator  should  try  to  think 
in  terms  of  how  the  learner  will  perceive  the  instructional  product.  Thus  the  evaluation  should 
be  “learner-centric.”  For  example,  if  the  evaluator  is  trying  to  determine  if  all  necessary 
navigation  and  operation  controls  are  included  (Criterion  5.d),  he  or  she  should  ask  whether  the 
learner  can  access  and  use  the  controls  for  all  required  purposes,  not  whether  a  computer 
programmer  can  use  them.  Although  it  is  vitally  important  for  the  developer  to  understand  the 
purpose  of  the  course,  this  understanding  is  moot  if  it  is  not  communicated  to  the  learner. 

No  matter  how  well  instruction  is  designed,  it  still  must  be  delivered  to  learners  in  some 
manner.  When  computers  are  the  chosen  delivery  method,  it  is  the  user  interface  that  shapes  the 
quality  of  the  instructional  delivery.  Instruction  has  been  delivered  using  computers  for  decades. 
During  this  time,  many  insights  have  been  developed  about  the  design  of  user  interfaces.  Table 
8  summarizes  ten  interface  design  heuristics  that  are  believed  to  enhance  the  usability  of  human- 
computer  interfaces  (Nielson,  1994).  Most  of  these  can  be  adapted  or  modified  to  support 
computer-  and  web-delivered  instruction.  Course  developers  and  managers  should  ensure  that 
they  consult  the  guidance  provided  by  experts  in  the  area  of  human-computer  interface  design. 
Three  other  classics  in  the  field  are:  Nielson  (1993),  Norman  (1988),  and  Shneiderman  (1998). 
Some  additional  usability  principles  are  recommended  by  Molich  and  Nielsen  (1990)  and 
Nielsen  (1993).  These  principles  include: 

1 .  Simple  and  natural  dialogue.  Dialogues  should  not  contain  information  that  is  irrelevant 
or  rarely  needed.  Every  extra  unit  of  information  in  a  dialogue  competes  with  the 
relevant  units  of  information  and  diminishes  their  relative  visibility.  All  information 
should  appear  in  a  natural  and  logical  order. 

2.  Speak  the  users  ’  language.  The  dialogue  should  be  expressed  clearly  in  words,  phrases, 
and  concepts  familiar  to  the  user,  rather  than  in  system-oriented  terms. 

3.  Minimize  the  users  ’  memory  load.  The  user  should  not  have  to  remember  information 
from  one  part  of  the  dialogue  to  another.  Instructions  for  use  of  the  system  should  be 
visible  or  easily  retrievable  whenever  appropriate. 

4.  Feedback.  The  system  should  always  keep  users  informed  about  what  is  going  on, 
through  appropriate  feedback  within  reasonable  time. 

5.  Clearly  marked  exits.  Users  often  choose  system  functions  by  mistake  and  will  need  a 
clearly  marked  “emergency  exit”  to  leave  the  unwanted  state  without  having  to  go 
through  an  extended  dialogue. 
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6.  Shortcuts.  Accelerators — unseen  by  the  novice  user — may  often  speedup  the  interaction 
for  the  expert  user  such  that  the  system  can  cater  to  both  inexperienced  and  experienced 
users. 


Table  8 

Computer  Interface  Usability  Heuristics 

(Adapted  from  Nielsen,  1994) 


Ensures  visibility  of  system  status 

The  software  keeps  the  user  informed  about  what  is 
going  on  through  appropriate  and  timely  feedback 
(e.g.,  “clicking”  sound  to  indicate  an  action  has  been 
completed). 

Maximizes  match  between  the  system 
and  the  real  world 

The  software  speaks  the  users’  language  rather  than 
jargon.  Information  appears  in  a  natural  and  logical 
order. 

Maximizes  user  control  and  freedom 

Users  are  able  to  exit  locations  and  undo  mistakes. 

Note:  learners  should  not  be  able  to  change  the 
instructional  sequence,  especially  for  novice  learners. 
More  experience  learners  or  those  taking  refresher 
courses  may  be  allowed  more  control. 

Maximizes  consistency  and  matches 
standards 

Users  do  not  have  to  wonder  whether  different  words, 
situations  or  actions  mean  the  same  thing.  Common 
operating  system  standards  are  followed. 

Prevents  errors 

The  design  provides  guidance,  which  reduces  the  risk 
of  user  errors. 

Supports  recognition  rather  than  recall 

Objects,  actions  and  options  are  visible.  The  user  does 
not  have  to  rely  on  memory.  Information  is  visible  or 
easily  accessed  whenever  appropriate. 

Supports  flexibility  and  efficiency  of 
use 

The  software  allows  experienced  users  to  use  shortcuts 
and  adjust  settings  to  suit. 

Uses  aesthetic  and  minimalist  design 

The  software  provides  an  appealing  overall  design  and 
does  not  display  irrelevant  or  infrequently  used 
information. 

Helps  users  recognize,  diagnose  and 
recover  from  errors 

Error  messages  are  expressed  in  plain  language,  clearly 
indicate  the  problem  and  recommend  a  solution. 

Provides  help  and  documentation 

The  software  provides  appropriate  online  help  and 
documentation,  which  is  easily  accessed  and  related  to 
the  users’  needs. 
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Chapter  7: 

Navigation  and  Operation  Criteria 
(Evaluation  Area  5) 

Navigation  refers  to  two  aspects  of  how  the  learner  relates  to  a  course:  1)  his  or  her  perceived 
ability  to  move  through  the  course  in  an  intentional  manner,  and  2)  the  learner’s  orientation,  or 
the  degree  the  learner  knows  where  he  or  she  is  in  the  course  and  how  to  get  to  another  part  of  it. 
Navigation  should  be  designed  to  support  the  delivery  of  instruction  in  the  sequence  and  manner 
that  is  consistent  with  the  purposes  and  instructional  objectives  of  the  course.  Navigation  should 
never  be  difficult  for  or  distracting  to  the  learner.  Once  the  learner  has  reached  the  desired 
location,  he  or  she  may  need  to  operate  the  controls  necessary  to  access  a  content  delivery 
medium  (e.g.,  play  a  video  or  start  an  animation).  The  operation  controls  should  be  easy  to  use 
to  avoid  distracting  the  learner. 

5.  a.  User  Interface  Makes  Course  Structure  Explicit 

The  user  interface  is  the  “portal”  that  lets  the  learner  into  the  course.  Its  design  can  support 
the  instructional  objectives  of  the  course  by  reflecting  and  supporting  the  structure  of  the  course. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  well  the  design  of  the  user  interface  reflects  the  structure  of 
the  course  and  helps  the  learner  understand  that  structure.  Use  the  anchor  statements  below  the 
scale  and  the  list  of  examples  to  help  assign  a  rating  on  this  criterion.  Be  cautious  when  rating  a 
course  that  allows  complete  learner  movement,  especially  for  introductory  courses.  Movement 
between  course  segments  should  only  be  allowed  when  it  is  instructionally  useful  (e.g.,  to  review 
material).  Additional  details  are  provided  in  the  discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

The  user  interface  is  the  mechanism  which  allows  the  computer  user  to  interact  with  the 
computer  program.  A  Graphical  User  Interface  (GUI)  is  a  type  of  interface  that  enables  the  user 
to  interact  with  representations  of  programs,  files,  and  options  by  means  of  icons,  menus,  and 
dialog  boxes  on  the  screen.  A  computer  user  directly  interacts  with  a  GUI  by  selecting  items  on 
the  screen  using  an  input  device  (e.g.,  keyboard,  mouse,  touch  screen,  etc.).  The  type  of 
interaction  required  must  be  considered  when  designing  the  GUI’s  appearance  and  functionality. 
For  example,  if  a  pointing  device  (e.g.,  mouse)  is  required,  the  layout  and  relative  locations  of 
items  on  the  screen  should  accommodate  this  functionality  so  the  user  can  recognize  selectable 
items  and  easily  select  them.  Selectable  items  need  to  be  large  enough  to  read  and  recognize  and 
spaced  enough  to  allow  the  user  to  select  one  without  affecting  another  item  on  the  screen.  In 
computer-based  instructional  applications,  the  learner  should  be  able  to  access  segments  of  the 
course  or  information  about  the  course  without  having  to  work  with  special  commands. 
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An  easy  to  use  interface  is  often  referred  to  as  “transparent.”  Transparency  “refers  to  the 
intuitiveness  of  the  interface  to  its  user”  (Tessmer,  1995,  p.  132).  A  transparent  interface  allows 
users  to  interact  directly  with  the  system,  not  via  an  intermediary.  The  interface  is  thus 
“transparent”  so  the  user  can  focus  on  the  task  at  hand  rather  than  on  the  interface  itself.  “When 
an  interactive  system  is  well  designed,  it  almost  disappears,  enabling  the  users  to  concentrate  on 
their  work  or  pleasure”  (Shneiderman,  1987,  p.  9). 

Several  techniques  have  been  used  to  design  transparent  GUIs.  One  technique  is  to  use 
perceptual  grouping  principles,  as  demonstrated  by  Gestalt  psychologists  (e.g.,  Weiten,  2004,  pp. 
143-146).  Humans  tend  to  perceive  items  as  belonging  together  if  they  share  characteristics  in 
common  (e.g.,  size,  shape,  color,  theme,  or  proximity).  Nielsen  (1993,  p.  117)  recommends 
organizing  interface  items  based  on  these  Gestalt  principles.  Such  organization  (grouping)  of 
items  on  the  display  affects  the  visual  focus  of  the  user  and  can  help  to  perceptually  establish  the 
structure  of  the  interface  (Brinck,  Gergle,  &  Wood,  2002,  pp.  192-197).  Brinck  and  others 
(2002,  pp.  194-197)  also  suggest  that  one  can  take  advantage  of  Gestalt  grouping  principles  by 
reversing  them  to  create  perceptual  boundaries  between  the  items  that  should  be  grouped 
together  and  others  that  belong  to  a  different  set.  For  example,  by  applying  a  different 
background  color  behind  menu  options,  the  menu  will  stand  out  from  the  rest  of  the  content  on 
the  page. 

All  computer  programs  require  a  well-designed  interface.  However,  instructional  programs 
need  to  go  beyond  just  a  transparent  interface.  The  design  of  the  interface  can  actually  support 
learning.  For  instructional  programs,  a  good  interface  design  reduces  the  cognitive  load  of  the 
learner  by  removing  frustrating  distractions  that  may  redirect  the  learner’s  attention  from  the 
instructional  content.  They  can  also  be  designed  to  help  the  learner  understand  the  structure  of 
the  course  and  how  its  various  segments  support  one  another. 

Various  techniques  have  been  used  to  organize  instructional  content  and  represent  it  on  the 
user  interface,  such  as  task-oriented,  topic-oriented,  or  chronologically-oriented.  The 
organization  scheme  should  remain  consistent  throughout  the  course  (Rosenfeld  &  Morville, 

1998;  also  see  Criteria  5.e,  5.f,  and  5.g).  It  is  also  important  to  keep  the  organization  simple, 
without  too  much  information  crammed  onto  each  screen  (Brinck,  et  al.,  2002,  p.  180). 

One  way  to  help  ensure  that  the  interface  design  has  good  usability  and  communicates  well  to 
the  user  is  to  describe  the  interface  in  terms  of  a  familiar  metaphor  or  analogy  (Eberts,  1994,  pp. 
143-145;  Neilsen,  1993,  pp.  126-129).  For  example,  one  familiar  metaphor  is  the  desktop 
interface.  In  this  design,  the  computer  desktop  is  based  on  artifacts  found  in  a  typical  office, 
including:  files,  folders,  and  a  recycling  bin.  Another  useful  metaphor  in  computer-delivered 
instruction  is  the  VCR  metaphor  for  controlling  videos  in  the  course.  In  this  case,  the  controls 
resemble  those  of  a  standard  VCR  (e.g.,  play,  pause,  fast  forward,  rewind,  etc.). 

When  choosing  a  metaphor  or  analogy,  it  is  important  to  choose  one  that  is  clear  and  that  will 
be  familiar  to  most  (if  not  all)  users.  The  intention  of  the  metaphor  is  to  augment  the  usability  of 
the  system  by  better  informing  users  about  how  the  system  works  (Nielsen,  1993,  pp.  128-129; 
Eberts,  1994,  Chapter  10).  “For  an  interface  metaphor  to  be  successful,  it  must  be  both  usable 
and  informative.  It  can  only  be  useful  if  its  imagery  matches  the  learner’s  prior  knowledge  and 
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experience  with  the  metaphor  as  well  as  with  the  multimedia  system’s  functionality”  (Tessmer, 
1995,  p.  134). 

Shneiderman  (1998,  pp.  204-205)  explains  that  good  use  of  a  metaphor  helps  to  prevent  the 
user  from  over  or  under-extension  of  the  system’s  capabilities  relative  to  the  user’s  mental 
models  and  expectations  of  what  the  system  can  do.  Good  metaphors  improve  the  user’s  ability 
to  learn  and  remember  how  to  interact  with  the  system,  whereas  a  system  with  no  metaphor  (or  a 
poorly  designed  one)  requires  the  user  to  retain  in  memory  the  meaning  of  each  icon  or  term 
used  in  the  system. 

Nielsen  (1993,  pp.  43-48)  recommends  that  interfaces  should  be  designed  to  support  both 
novice  and  expert  users.  He  explains  that  there  are  three  dimensions  that  explain  how  users 
differ:  1)  computer  experience  in  general;  2)  knowledge  of  the  task  domain  or  topic;  and  3) 
experience  using  a  specific  interface.  Due  to  the  variability  in  interface  designs,  expertise  in  one 
interface  may  or  may  not  be  helpful  when  using  a  new  interface.  This  is  especially  true  for 
computer-delivered  instruction.  Even  computer  experts  may  not  have  any  experience  in  a 
specific  content  domain  and  may  require  help  to  understand  the  structure  of  the  course.  This  is 
one  reason  why  every  instructional  application  should  include  a  tutorial  and  a  help  function  that 
explain  all  controls  and  capabilities  (see  Criteria  5.b  and  5.c  below). 

Content  Mans.  A  technique  that  can  help  make  the  organization  of  the  content  apparent  to  the 
learner  is  to  use  a  content  map  or  a  site  map  (Ross,  1998).  This  can  be  in  the  form  of  an  outline, 
showing  major  segments,  lessons,  or  topics  and  how  they  relate  to  one  another.  This  is  much 
like  a  table  of  contents  in  a  book.  However,  as  discussed  in  Criterion  l.a,  developers  must  take 
care  that  learners  (especially  introductory  learners)  be  guided  through  the  course  rather  than  be 
allowed  to  browse  topics  in  any  order  (Jones  and  Okey,  1995).  For  learners  who  are  taking 
refresher  training  or  those  that  want  to  review  a  previously  completed  course,  more  freedom  to 
select  topics  can  be  provided. 

Managing  Complex  Content.  Complex  content  may  sometimes  overload  the  cognitive 
processing  capabilities  of  the  learners.  This  is  especially  true  when  various  presentation  media 
and  modalities  are  combined.  Ross  (1998)  suggests  several  tools  that  can  help  learners  to 
manage  content: 

1.  Glossary:  definitions  of  technical  terms,  links  to  original  location  in  the  course  to  provide 
context;  perhaps  a  “pronounce”  button  to  demonstrate  correct  pronunciation  of  terms. 

2.  Index:  list  and  brief  overview  of  the  major  topics;  “tracking”  function  to  show  what  parts 
of  the  course  have  been  completed  (This  is  similar  to  the  content  or  site  maps  discussed 
in  sections  l.g.  and  5.f.) 

3.  Bookmarking:  to  allow  learners  to  return  to  the  area  of  the  course  they  last  visited 

4.  Searching:  to  help  learners  find  specific  information  quickly  and  efficiently 

5.  Detailed  resource  lists:  categorized  by  subject  area. 

6.  Notebook:  An  electronic  resource  with  a  print  and  edit  feature  to  allow  learners  to 
summarize  and  paraphrase  course  material.  A  float,  pop-up  window  could  be  used  to 
allow  learners  to  move  through  course  pages,  while  still  having  access  to  the  notebook. 

7.  On-line  help:  In  addition  to  a  built-in  help  function  (see  section  5.a),  on-line  help  can 
allow  learners  to  ask  detailed  questions  about  course  operation  or  content.  This  could 
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also  include  a  course  facilitator’s  e-mail  address,  but  care  must  be  taken  to  provide  timely 
responses  to  questions. 


5.b.  Tutorial  is  Available  to  Explain  Navigation  and  Operation  Features 

A  tutorial  is  a  “mini”  course  that  provides  instruction  on  how  to  use  the  navigation  and 
operation  features  to  optimize  the  learning  experience. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  many  of  the  navigation  and  operation  features  are  covered 
by  a  tutorial  to  help  the  learner  to  understand  how  to  use  the  navigation  and  operation  features. 
The  anchor  statements  and  list  of  ways  to  design  tutorials  should  be  used  to  help  assign  a  rating 
on  this  criterion.  Additional  information  on  this  criterion  can  be  found  in  the  discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

Prior  to  beginning  a  course,  learners  should  be  provided  with  a  tutorial  that  explains  all  of  the 
navigational  features.  The  tutorial  should  also  explain  how  the  learner  should  set  up  his  or  her 
computer  to  optimally  view  text,  graphics,  videos,  and  other  presentation  formats  (see  Criterion 
7.c). 

The  tutorial  should  provide  “hints”  and  suggestions  for  how  to  use  the  power  of  each 
navigational  feature  as  support  for  the  instruction.  The  tutorial  should  also  provide  guidance  on 
the  appropriate  setup  of  the  computer  (e.g.,  level  of  screen  resolution). 

Some  possible  ways  to  design  tutorials  include: 

•  Walk-through  with  text  and  narration 

•  Animation  of  moving  cursor  with  text  narration 

•  Animated  “guide”  (e.g.,  avatar) 

•  Text  and  graphics 


5,c.  Help  Function  is  Available  to  Explain  Navigation  and  Operation  Features 

A  help  function  is  a  mechanism  that  allows  a  learner  to  seek  information  on  the  use  of  a 
specific  navigation  or  operation  feature. 

Using  the  Scale 

This  scale  is  used  to  evaluate  the  course  on  the  presence  and  usefulness  of  a  help  function. 

Use  the  anchor  statements  and  list  of  ways  to  access  help  as  guides  in  assigning  a  rating  on  this 
criterion.  Additional  information  on  this  criterion  is  provided  in  the  discussion  below. 
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Terms.  Concepts.  Techniques,  and  Supporting  Literature 

A  help  feature  should  always  be  available  if  a  learner  forgets  how  or  when  to  use  a  given 
feature.  Replaying  short  segments  from  the  tutorial  might  provide  this  targeted  help. 

Ross  (1998)  suggests  that  an  on-line  help  function  can  allow  learners  to  ask  detailed  questions 
about  course  operation  or  content.  This  could  also  include  a  course  facilitator’s  e-mail  address. 
Care  must  be  taken  to  provide  timely  responses  to  e-mail  questions  lest  learners  become 
frustrated  and  lose  interest  in  the  course.  An  alternative  might  be  a  “call-in”  help  desk. 

However,  a  help  desk  must  be  maintained  and  monitored  as  long  as  the  course  is  in  use. 


5.d.  Includes  all  Necessary  Navigation  and  Operation  Controls. 

The  learner  needs  to  be  able  to  move  through  the  course  as  efficiently  as  possible. 
Furthermore,  when  arriving  at  the  desired  location,  he  or  she  should  be  able  to  easily  operate  any 
controls  needed  to  interact  with  the  course  (e.g,,  play  a  video,  answer  a  question,  interact  with  a 
simulation,  etc.). 

Using  the  Scale 

This  scale  is  used  to  rate  the  course  in  terms  of  the  number  of  navigation  and  operation 
controls  provided  in  the  course,  as  well  as  the  degree  of  control  provided  to  the  learner.  The 
anchor  statements  and  list  of  navigation  and  operation  controls  found  below  the  scale  should  be 
used  as  guides  for  assigning  a  rating.  Consult  the  discussion  below  for  additional  information  on 
this  criterion. 

Terms,  Concepts.  Techniques,  and  Supporting  Literature 

When  it  is  appropriate  for  the  learner  to  control  navigation  or  the  operation  of  a  delivery 
medium  (e.g.,  video)  a  mechanism  to  access  these  features  should  be  provided.  This  access  can 
be  made  available  in  various  ways  (e.g.,  buttons,  highlighted  text,  voice  commands,  etc.),  but 
whichever  method  is  selected,  it  should  remain  consistent  throughout  the  course.  Table  9  is  a  list 
of  possible  navigation  and  operation  controls  with  a  short  description  of  their  function.  In 
general,  most  of  these  controls  should  be  included  in  any  computer-delivered  course. 

As  discussed  under  the  topic  of  content  presentation  (Criterion  1  .a),  learner  control  is  a 
double-edged  sword.  Too  much  control  can  be  detrimental  to  effective  learning  (Clark,  R.,  2003, 
p.  14;  Niemiec,  et  al.,  1996),  but  some  level  of  control  can  be  beneficial  (Doherty,  1998). 
Providing  the  ability  to  control  pacing  and  movement  within  a  lesson  (e.g.,  replaying  a  section  or 
controlling  a  video  presentation)  is  a  positive  design  goal.  However,  in  most  cases,  the  structure 
and  order  of  instructional  content  should  not  be  left  in  the  hands  of  the  learner.  In  the  context  of 
this  criterion,  “necessary”  controls  are  those  that  support  the  navigation  through  the  course  and 
allow  control  of  content  presentation.  For  example,  “necessary”  controls  for  a  video 
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presentation  are  those  that  allow  the  learner  to  access  and  control  the  video  (e.g.,  start,  stop, 
rewind,  or  pause).  If  there  are  no  video  presentations,  these  controls  are  not  necessary. 


Table  9 

Possible  Navigational  and  Operation  Controls 
in  Computer-Delivered  Courses 


Control  Name 

Description  of  Function 

Start 

Used  to  begin  the  course  or  a  section  within  the  course.  Can  also  be 
used  to  begin  a  video  or  audio  segment  or  begin  an  interactive  exercise. 

Stop 

Used  to  end  a  section  or  any  activity  within  a  section  (e.g.,  a  video). 

Pause 

Used  to  suspend  (pause)  an  ongoing  segment  of  the  course.  This  might 
be  narration,  a  video,  or  an  interactive  exercise. 

Resume 

Used  to  restart  (resume)  a  paused  activity.  In  some  cases  this  is  done  by 
hitting  the  “start”  button  again. 

Go  Forward  (by  screen) 

Used  to  move  to  the  next  screen  of  instruction. 

Go  Backward  (by  screen) 

Used  to  move  back  to  the  previous  screen  of  instruction. 

Go  Forward  (by  lesson) 

Used  to  move  forward  to  the  next  lesson. 

Go  Backward  (by  lesson) 

Used  to  move  back  to  the  previous  lesson. 

Return  to  Beginning 

Used  to  return  to  the  beginning  of  the  course. 

Return  to  Main  Menu 

Used  to  go  to  the  main  menu  of  the  course  (table  of  contents  or  lesson 
outline) 

Bookmark 

Used  to  mark  the  learner’s  place  in  the  course  so  he  or  she  may  return  to 
that  specific  screen. 

Print  Screen 

Used  to  print  the  current  screen. 

Print  Selected  Text 

Used  to  print  a  selected  segment  of  the  text.  Some  courses  also  allow 
the  learner  to  print  the  entire  text  and/or  narration  script. 

Take  Notes 

Enables  the  learner  to  take  notes  or  make  comments  about  the  current 
instruction. 

Save  Copy 

Enables  the  learner  to  save  and  retrieve  any  notes  or  activities  (e.g., 
computations)  made  during  the  course. 

Volume  control  for  audio 

Used  to  adjust  the  volume  for  audio  presentations. 

VCR-type  controls  (for 
videos) 

Used  to  control  any  videos  used  in  the  course.  Should  resemble 
controls  on  standard  Video  Cassette  Recorders  (VCRs).  Should 
include:  start,  stop,  pause,  rewind,  fast  forward,  etc. 

5.e.  Navigation  and  Operation  Controls  are  Clearly  and  Consistently  Labeled 


Using  the  Scale 

This  scale  is  used  to  rate  the  course  on  how  many  of  the  navigation  and  operation  controls  are 
clearly  and  consistently  labeled.  Use  the  anchor  statements  below  the  scale  to  help  assign  your 
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rating  on  this  criterion.  Additional  information  on  this  criterion  is  provided  in  the  discussion 
below. 

Terms.  Concepts,  Techniques,  and  Supporting  Literature 

Consistency  is  essential  to  establish  commonality  and  unity  across  the  various  content  pages 
of  the  course  (Brinck,  et  al.,  2002,  pp.  1 10-1 12;  Nielsen,  1993,  pp.  132-134).  Gestalt 
psychologists  have  demonstrated  that  humans  perceive  items  to  be  grouped  together  if  they  are 
located  close  to  one  another  and  share  common  features  (e.g.,  colors,  shapes,  font  styles,  etc.; 
Weiten,  2004,  pp.  143-146).  Following  these  principles,  the  grouping  of  navigation  and 
operation  controls  can  help  the  learner  understand  their  functions  and  prepare  for  taking  actions 
(Adam,  Hommel,  &  Umilta,  2003;  Borchers,  Deussen,  Klingert,  &  Knorzer,  1996). 

Fitts  (1954)  demonstrated  that  the  rate  of  performance  in  a  given  task  is  approximately 
constant  over  a  range  of  movement.  However,  the  performance  rate  falls  off  outside  the  optimal 
range  of  movement.  Thus,  grouping  controls  not  only  helps  the  learner  understand  their 
function,  such  grouping  can  also  help  in  the  speed  and  accuracy  of  actions  (Eberts,  1994; 
Shneiderman,  1998). 

Whichever  design  is  chosen  for  navigation  controls,  each  should  be  clearly  and  consistently 
labeled.  In  no  case  should  the  appearance  or  label  of  a  control  be  different  in  different  course 
segments.  Consistency  is  regarded  as  one  of  the  most  important  usability  issues  in  any  computer 
interface  (Molich  and  Nielsen,  1990).  This  is  true  for  this  criterion  in  terms  of  consistency  of 
labeling.  It  is  also  true  for  the  next  two  criteria,  consistency  of  location  (5.f)  and  consistency  of 
operation  (5.g). 

Designers  should  consider  using  standard  browser  controls  as  much  as  possible.  Standard 
browser  controls  are  those  that  are  used  by  almost  all  software  that  allows  users  to  access  the 
internet.  Most  computer  users  are  familiar  with  how  these  controls  are  labeled,  so  they  should  be 
able  to  more  easily  adapt  to  their  use  in  instructional  applications.  Even  these  standard  controls 
should  be  clearly  labeled  (e.g.,  easily  seen).  When  additional  controls  are  needed,  they  should 
also  be  clearly  labeled  and  these  labels  should  be  consistent. 

5,f.  Navigation  and  Operation  Controls  are  Located  in  a  Consistent  Place 

Just  as  navigation  and  operation  controls  should  have  a  consistent  appearance,  they  should 
also  be  located  in  a  consistent  place  on  the  computer  screen. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  consistently  navigation  and  operation  controls  are  located 
on  the  computer  screens  throughout  the  course.  Use  the  anchor  statements  below  the  scale  as 
guides  in  assigning  a  rating  on  this  criterion.  The  discussion  below  provides  additional 
information  on  this  criterion. 
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Terms.  Concepts,  Techniques,  and  Supporting  Literature 

It  is  often  useful  to  cluster  navigational  controls  in  a  single  “control  panel,”  but  this  panel 
should  always  be  found  in  the  same  place.  The  specific  place  is  not  as  important  as  maintaining 
a  consistent  location.  Operation  controls  should  also  be  located  in  a  consistent  place.  For 
example,  if  videos  are  used  in  the  course,  the  controls  should  be  located  near  the  video  display 
window  (usually  under  the  window). 

The  Gestalt  grouping  principles  discussed  under  consistent  labeling  of  controls  (Criterion 
5.e.)  also  support  this  criterion  (Weiten,  2004).  The  research  conducted  by  Adam  and  his 
colleagues  (2003)  demonstrated  the  close  association  between  perception  and  action.  Once  the 
learner  has  learned  to  deal  with  common  controls  in  a  certain  location,  a  change  in  this  location 
would  require  relearning  and  would  likely  be  a  distraction  from  the  learning  experience. 


5.g.  Navigation  and  Operation  Controls  Operate  Consistently 


Using  the  Scale 

This  scale  is  used  to  evaluate  how  consistently  the  controls  function  in  different  places  in  the 
course.  Use  the  anchor  statements  below  the  scale  to  assist  in  assigning  a  rating  on  this  criterion. 
Additional  information  is  provided  in  the  discussion  below. 

Terms.  Concepts.  Techniques,  and  Supporting  Literature 

Consistency  of  function  is  also  very  important  for  navigation  and  operation  controls.  The 
learner  should  understand  exactly  what  will  happen  any  time  he  or  she  activates  a  navigational 
control  and  the  resulting  action  should  operate  reliably.  It  is  also  important  that  the  interface 
provide  a  visual  or  audio  cue  that  the  program  has  registered  his  or  her  choice  and  that  the  action 
is  taking  place  or  the  learner  is  going  someplace  (Jones  and  Okey,  1995,  citing  Nicol,  1990). 

Consistency  in  the  actions  of  navigation  and  operation  controls  is  just  as  important  as 
consistency  in  the  location  of  controls  (Shneiderman,  1998  p.  74).  As  mentioned  in  the 
discussion  on  location  consistency,  functional  consistency  can  also  be  achieved  by  using 
standard  browser  buttons  as  often  as  possible. 


5.h.  Course  is  Designed  to  Show  Learner’s  Location 

As  the  learner  moves  through  a  course,  he  or  she  can  benefit  from  any  cues  that  help  integrate 
the  many  separate  items  of  information  needed  to  be  learned.  Knowing  his  or  her  current 
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location  and  how  he  or  she  arrived  there  (see  Criterion  5.i)  can  be  powerful  tools  to  assist  the 
learner  in  this  integration.  This  criterion  and  the  next  evaluate  these  two  capabilities. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  easily  a  learner  can  determine  his  or  her  location  in  the 
course.  Use  the  anchor  statements  and  the  list  of  example  techniques  found  below  the  scale  to 
help  assign  a  rating.  Additional  information  is  provided  in  the  discussion  below. 

Terms.  Concepts.  Techniques,  and  Supporting  Literature 

Utting  and  Yankelovich  (1989)  report  that  computer  users  frequently  complain  about  being 
lost  in  an  interactive  program.  A  learner  can  waste  valuable  instructional  time  trying  to 
determine  his  or  her  location  in  the  course.  This  may  even  lead  to  a  loss  of  the  motivation  to 
continue  the  instruction. 

As  discussed  under  content  presentation  (Criterion  l.a),  learners  should  be  provided  with  a 
content  map  (site  map)  or  other  aid  that  helps  them  understand  where  they  are  in  a  course  and 
how  they  will  proceed.  According  to  Ross  (1998),  a  site  map  is  a  detailed  hierarchical  structure 
of  the  course  with  the  following  features: 

•  It  allows  learners  to  quickly  obtain  an  overview  of  the  course’s  structure. 

•  It  should  ideally  be  available  from  every  page. 

•  It  should  show  where  the  learner  is  currently  located  (e.g.,  by  highlighting  the  current 
screen  on  the  map). 

•  It  should  help  the  learner  “track”  his  or  her  progress  by  showing  completed  topics. 


5.i.  Course  is  Designed  to  Show  How  Learner  Arrived  at  Location 

As  mentioned  above,  knowing  how  he  or  she  moved  from  topic  to  topic  to  arrive  at  a  given 
location  can  help  the  learner  to  understand  the  interrelationship  among  various  items  of 
information. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  easy  it  is  for  the  learner  to  understand  how  he  or  she 
arrived  at  their  current  location  in  the  course.  Use  the  anchor  statements  and  list  of  techniques 
found  below  the  scale  to  help  assign  a  rating  on  this  criterion.  Additional  information  is 
provided  in  the  discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

As  important  as  knowing  where  they  are  currently  located  in  the  course  (Criterion  5.h), 
learners  should  also  understand  the  instructional  path  that  led  them  to  their  current  location. 
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This  will  help  them  integrate  the  new  instructional  information  with  their  previously  developed 
knowledge  structures.  This  can  also  help  learners  get  back  to  previous  locations  when  they  wish 
to  review  material. 

5.i.  Course  is  Designed  to  Show  Estimated  Time  Required  for  Each  Module 


Using  the  Scale 

This  scale  is  used  to  evaluate  how  often  time  estimates  for  completing  course  modules  are 
provided.  Use  the  anchor  statements  and  list  of  example  techniques  below  the  scale  to  assist  in 
assigning  a  rating  on  this  criterion.  Additional  information  is  provided  in  the  discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

One  of  the  strengths  of  computer-based  or  web-based  training  is  the  capability  for  learners  to 
take  the  training  at  their  job  sites  or  on  their  own  time.  They  may  want  to  take  the  training 
during  slack  times,  break  times,  or  lunch.  They  need  to  know  how  long  it  will  take  to  complete  a 
module  in  order  to  schedule  their  training  time.  In  most  cases,  it  is  important  for  learners  to 
complete  a  module  and  the  associated  assessment  and  feedback  before  taking  a  break  from  the 
course.  Therefore  the  course  designer  should  provide  estimated  times  for  each  module  to 
facilitate  scheduling.  These  time  estimates  should  be  derived  from  formative  evaluations. 

One  method  to  show  the  learners  the  estimated  time  to  complete  a  module  is  to  provide  a 
statement  of  this  estimated  time  at  the  beginning  of  each  module.  Another  method  is  to  use  a  bar 
diagram  or  some  other  symbolic  indicator  of  how  much  of  the  module  has  been  completed. 


A 
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Chapter  8: 

Content  Presentation 
(Evaluation  Area  6) 

The  user  interface  is  the  mechanism  for  presentation  of  the  instructional  content.  How  the 
various  methods  of  presentation  are  used  can  either  support  or  distract  from  the  instruction.  The 
following  criteria  are  used  to  help  ensure  that  the  presentation  supports  the  instruction. 

6,a,  There  are  No  Sensory  Conflicts 

Using  a  variety  of  delivery  modalities  (e.g.,  video,  graphics,  audio)  can  make  a  course  more 
engaging  and  more  informative.  However,  if  there  are  conflicts  between  different  modalities, 
learning  effectiveness  can  suffer.  Sensory  conflicts  occur  when  different  content  is  presented 
simultaneously  in  two  or  more  modalities  or  when  the  same  content  is  not  properly  synchronized 
across  modalities. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  often  sensory  conflicts  are  encountered  when  the  course 
delivers  the  content  using  multiple  modalities  (e.g.,  audio  and  video).  Use  the  anchor  statements 
and  examples  of  sensory  conflicts  found  below  the  scale  to  assist  in  assigning  a  rating  on  this 
criterion.  Additional  information  is  provided  in  the  discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

Some  cognitive  psychologists  (e.g.,  Baddeley,  1992;  Paivio,  1990;  Penny,  1989)  theorize  that 
humans’  working  memory  includes  separate  processes  for  different  sensory  modalities  (e.g., 
auditory,  vision,  etc.).  According  to  these  theorists,  the  separate  processes  have  separate 
amounts  of  processing  resources  and  can  either  reinforce  or  inhibit  each  other. 

Course  designers  often  try  to  use  a  variety  of  presentation  media  (audio,  text,  graphics,  or 
video)  to  present  instructional  content.  This  is  beneficial  only  if  the  media  work  with  one 
another  in  a  supportive  fashion.  If  they  do  not,  they  can  become  distractions.  A  learner  cannot 
simultaneously  attend  to  multiple  presentations  of  different  information.  Nor  can  the  learner 
determine  which  medium  to  attend  to  or  which  information  is  most  important.  For  example,  if 
using  audio  to  supplement  textual  presentation  of  content,  the  two  media  should  present  the  same 
information.  This  does  not  mean  word-for-word  duplication.  However,  reinforcing  important 
points  in  the  narration  by  using  textual  bullets  may  improve  learning. 

Some  research  indicates  that  people  learn  better  when  narration  is  supplemented  with 
pictures,  videos,  illustrations,  or  animations  (Mayer,  2003).  Other  research  has  shown  that 
combining  audio  explanations  with  visual  instructions  can  have  negative  consequences  due  to 
this  concurrent  information  causing  cognitive  overload  (Kalyuga,  2000).  This  overload  can  be 
avoided  if  the  concurrent  instructional  messages  do  not  contain  extraneous  words,  pictures,  or 
sounds  (Mayer,  2003). 
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In  general,  research  has  shown  that  conflicting  auditory  messages  should  not  be  provided 
simultaneously  (Cherry,  1953;  Gray  &  Wedderbum,  1960;  Moray,  1959;  Moray,  Bates,  & 
Barnett,  1965;  Treisman,  1964).  For  example,  Moreno  and  Mayer  (2000a)  examined  the  impact 
on  performance  of  adding  background  sounds  and  music  to  a  multimedia  instructional  message. 
While  reviewing  instructional  material  on  the  formation  of  lightning,  learners  in  different  groups 
received  background  music,  environmental  sounds,  both,  or  neither.  After  completing  the 
instruction,  participants  completed  a  retention  task  that  asked  them  to  explain  the  lightning 
process  (e.g.,  “What  could  you  do  to  decrease  the  intensity  of  lightning?”).  Participants  who 
experienced  background  music  recalled  less  verbal  material  and  performed  worse  on  the  transfer 
test  than  those  who  experienced  no  background  music.  Participants  who  received  both  music 
and  sounds  performed  worse  than  all  the  other  three  groups.  The  results  of  a  second  experiment, 
teaching  the  operation  of  hydraulic  braking  systems,  were  similar  to  the  first  experiment.  The 
authors  conclude  that  auditory  adjuncts  can  overload  the  learner’s  auditory  working  memory  and 
are  detrimental  to  learning. 

It  is  difficult  to  attend  to  a  single  voice  when  multiple  voices  are  present.  Rudmann, 
McCarley,  and  Kramer  (2003)  examined  the  impact  of  presenting  a  video  image  of  a  target 
speaker  to  help  participants  identify  words  spoken  by  that  speaker.  In  their  study,  participants  in 
different  groups  listened  to  multiple  speakers  (a  target  speaker  and  2,  3,  or  4  distracting 
speakers).  Their  task  was  to  focus  on  the  target  speaker  and  identify  target  words  spoken  by  the 
target  speaker  while  trying  to  ignore  the  distracting  speakers.  Participants  in  each  group  either 
received  or  did  not  receive  a  video-displayed  image  of  the  target  speaker.  Results  indicated  that 
the  performance  of  the  participants  declined  when  the  number  of  distracting  speakers  increased. 
Furthermore,  the  participants’  “ability  to  track  target  words  declined  regardless  of  whether  or  not 
a  display  of  the  target  speaker  was  provided.  However,  when  the  video  display  was  not 
provided,  performance  declined  at  a  faster  rate”  (Rudmann,  et  al.,  2003,  p.  332).  These  results 
demonstrate  that  for  tasks  requiring  attention  to  multiple  speakers  (e.g.,  air  traffic  control), 
including  supporting  media  (e.g.,  video  images)  may  help  task  performance.  However,  for 
instructional  applications,  it  is  probably  better  to  use  only  one  speaker  at  a  time. 


6.b.  All  Media  are  Clear  and  Sharp 

No  matter  which  medium  is  used,  learners  cannot  attend  to  instructional  information  if  it  is 
presented  in  a  manner  that  does  not  allow  them  to  perceive  it. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  many  of  the  media  used  in  the  course  are  presented  in  a 
format  that  is  easily  perceived  by  the  learner  (e.g.,  they  are  clear  and  sharp).  Use  the  anchor 
statements  and  list  of  examples  of  clear  and  sharp  media  found  below  the  scale  to  help  assign  a 
rating  on  this  criterion.  Additional  information  is  provided  in  the  discussion  below. 
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Terms,  Concepts.  Techniques,  and  Supporting  Literature 

If  the  learner  cannot  understand  the  narration,  read  the  text,  or  see  the  important  portions  of  a 
graphic  or  video,  the  instructional  message  will  be  garbled.  Developers  should  always  ensure 
that  the  instructional  information  is  actually  being  conveyed  by  the  chosen  medium.  They 
should  also  ensure  that  the  media  do  not  become  distractions  (e.g.,  annoying  soundtracks, 
blinking  lights,  irrelevant  animations;  Nash,  2003).  Regardless  of  the  specific  medium,  it  is  a 
good  general  principle  to  try  to  avoid  the  need  for  the  learner  to  scroll  either  horizontally  or 
vertically. 

The  use  of  different  types  of  media  involves  different  issues.  The  following  discusses  some 
of  the  presentation  issues  for  different  media  types. 

Text.  Text  should  be  designed  to  enhance  readability  and  comprehension  (Bostock,  2004). 
Only  readable  (legible)  fonts  should  be  used  (e.g.,  those  that  are  large  enough  to  avoid  eye  strain 
and  are  displayed  clearly  on  the  computer  screen  at  the  recommended  level  of  resolution).  The 
tutorial  or  setup  module  (5.b)  should  include  guidance  for  setting  up  the  proper  screen  resolution 
and  the  setup  should  be  confirmed  with  an  “optimization”  test  (Criterion  7.c). 

Morkes  and  Nielsen  (1997;  1998)  maintain  that  persons  reading  web  pages  generally  prefer 
writing  that  is: 

•  Concise  (e.g.,  makes  its  points  quickly). 

•  Easy  to  scan  (e.g.;  uses  inverted  pyramid  style  like  news  articles  “in  which  the  news  and 

conclusions  are  presented  first,  followed  by  details  and  background  information”  [p.  13  of 

27]  and  uses  headings,  bold  face,  and  other  cues). 

•  Objective  (rather  than  promotional). 

They  found  that  rewriting  web  pages  combining  these  three  changes  resulted  in  124%  higher 
measured  usability  in  their  first  study  (1997)  and  159%  higher  usability  in  their  second  study 
(1998). 

Although,  text  has  been  the  primary  instructional  medium  for  centuries,  displaying  text  on  a 
computer  screen  can  cause  problems  (e.g.,  eye  strain).  During  the  initial  introduction  of 
computers,  it  was  generally  found  that  reading  efficiency  (e.g.,  reading  speed)  was  lower  from  a 
standard  computer  display  than  from  paper  (e.g.,  Mutter,  Latremouille,  Treumiet,  &  Beam, 

1982).  Neilson  (1997)  reports  that  most  people  only  read  50%  of  the  text  on  a  screen  and  at  a 
rate  about  25%  slower  compared  to  reading  the  same  text  on  paper.  He  also  maintains  that 
readers  tend  to  scan  rather  than  read  all  the  text  on  a  screen.  Therefore,  Nielson  (1997)  and 
many  others  (e.g.,  Thibodeau,  1997;  ANSI/  ADA,  2002)  recommend  limiting  the  amount  of  text 
on  a  screen. 

Computer  displays,  combined  with  various  enhancements  (e.g.,  double  spacing,  negative 
contrast,  bold  face,  etc.)  have  been  shown  to  reduce  the  performance  differences  between  readers 
of  paper-  and  computer-displayed  text.  Mutter  and  Maurutto  (1991)  examined  the  differences 
between  reader  performance  when  using  an  enhanced  computer  display  and  those  using  paper. 
They  found  no  differences  in  reading  speed  and  comprehension  between  the  two  groups.  They 
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concluded  that  computer  display  capabilities  and  presentation  techniques  were  now  mature 
enough  to  make  computer-displayed  text  a  viable  option. 

Using  larger  font  sizes  on  the  computer  display  is  often  preferred  by  the  reader.  For  example, 
Chen,  Jackson,  Parsons,  Sindt,  Summerville,  Tharp,  Ullrich,  and  Caffarella  (1996)  found  no 
differences  in  reading  comprehension  between  groups  presented  10,  12,  14,  or  16  point  fonts. 
However,  most  of  the  participants  preferred  either  the  14  or  16  point  fonts  (97.78%  and  83.78% 
respectively).  Similar  results  were  found  by  Pacheco,  Day,  Cribelli,  Jordan,  Murry,  and 
Persichitte  (1999).  They  found  that  readers  preferred  larger  fonts  and  double  spacing  in  both 
scrolled  and  not  scrolled  displays. 

Harpster,  Freivalds,  Shulman,  and  Leibowitz  (1989)  compared  the  search  speeds  for  persons 
reading  text  on  a  computer  or  on  paper.  The  participants  viewed  a  large  pattern  of  numbers  and 
were  required  to  locate  four  specified  numbers  in  this  pattern.  They  found  performance 
differences  between  the  computer  and  paper  conditions  when  a  low  resolution  computer  display 
was  used,  but  no  differences  when  the  text  was  shown  on  a  high  resolution  display. 

Garland  and  Noyes  (2004)  found  no  differences  in  the  test  performance  of  learners  who  read 
from  paper  or  a  computer  screen.  These  results  are  difficult  to  interpret  because  the  font  size  on 
the  computer  display  is  not  provided.  They  did  find  differences  in  the  way  participants  in  the 
two  groups  retrieved  knowledge.  Over  7  sessions,  the  computer  group  took  longer  to  shift  from 
episodic  memory  (“remember”)  to  semantic  memory  (“know”)  than  the  paper  group.  The 
authors  speculate  that  the  computer  presentation  may  interfere  with  the  learning  process. 
However,  this  conclusion  should  be  considered  very  tentative  due  to  the  lack  of  detail  provided 
on  experimental  procedures. 

Lee  and  Boling  (1999)  summarized  screen  design  guidelines  from  77  books  and  articles. 

Their  guidelines  on  typography  (pp.  10-21)  include: 

•  Be  consistent  in  the  use  of  textual  cues  and  signals  to  the  learner.  Consistency  in  these 
typographical  signals  can  establish  and  convey  clear  visual  messages  to  the  learners  that 
they  are  now  reading  a  certain  subject  or  section  of  the  course  content.  These  cues  can 
include  the  use  of  titles  and  subtitles  to  improve  comprehension  by  providing  structure 
and  to  signal  the  flow  of  ideas. 

•  Use  both  upper  and  lower  case  letters  (i.e.,  capitals  only  for  indicating  the  first  letter  of  a 
sentence  or  a  proper  name)  for  increased  legibility.  All  upper  case  characters  should  only 
be  used  occasionally  (e.g.,  for  purposes  of  emphasis). 

•  Use  high  contrast  between  letters  and  background. 

•  Left-justify  text,  but  do  not  right-justify  it. 

•  Limit  text  to  approximately  65  characters  per  line  (cpl),  or  a  maximum  of  8-10  words  per 
line.  Dyson  and  Hasselgrove  (2001)  confirmed  this  general  guideline.  They  found  that 
reading  comprehension  was  better  when  line  length  was  55cpl  as  compared  to  lOOcpl. 
Readability  can  also  be  improved  by  presenting  text  in  columns  that  are  no  wider  than  a 
person’s  comfortable  eye  span. 

•  Use  highlighting  techniques  conservatively  and  carefully  when  trying  to  attract  the 
learner’s  attention.  Blinking,  flashing,  or  the  use  of  reverse  video  (e.g.,  white  on  black) 
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can  be  used  to  attract  attention.  However,  overuse  of  these  techniques  can  interfere  with 
legibility. 

•  Select  a  font  with  a  simple,  clean  style  and  use  only  a  few  different  font  sizes  on  any  one 
screen. 

Thibodeau  (1997)  recommends  the  following  techniques  when  formatting  text  for  display  on 
a  computer  screen: 

•  Use  white  (blank)  space  to  separate  blocks  of  textual  information 

•  Convert  sentences  containing  serial  items  to  lists 

•  Use  upper  case  only  for  titles  and  emphasis 

•  When  using  highlighted  or  bold  text,  limit  this  to  10%  of  total  screen  text 

•  Text  should  never  “blink”  except  temporarily  to  attract  attention 

•  Use  no  more  than  one  attention-getting  technique  per  screen 

Nielson  (1997)  adds  the  following  recommendations  for  the  use  of  headings  to  help  the 
reader: 

•  Use  2  or  even  3  levels  of  headings 

•  Use  meaningful,  not  “cute”  headings  -  these  should  be  the  same  as  in  the  table  of 
contents 

•  Use  highlighting  or  other  methods  to  emphasize  the  headings  and  catch  the  learner’s  eye 

Here  are  a  few  additional  suggestions.  Moreno  and  Mayer  (2000b)  demonstrated  that 
instructional  text  written  in  first  and  second  person  resulted  in  better  learning  than  when  the  text 
was  written  in  a  more  impersonal  tone.  All  technical  words  should  be  explained  and  included  in 
an  easily  accessible  glossary  (Bostock,  2004).  A  balance  between  text  and  graphics  can  add  to 
the  aesthetic  appeal  of  the  course. 

Audio.  Ross  (1998)  suggests  three  ways  audio  can  be  used  to  support  instruction.  Audio 
messages  can  be  used  to  emphasize  important  points  or  serve  as  reminders  for  learners  (e.g., 
“Pay  particular  attention  to  sections  1-6.”).  Audio  lectures  can  be  used  to  present  short 
explanations  of  important  concepts.  Audio  can  also  be  used  to  pronounce  glossary  terms.  This 
can  be  very  effective  when  learners  are  not  fluent  in  English  or  are  not  familiar  with  technical 
terms.  All  audio  presentations  should  be  easily  understood.  Also,  some  research  indicates  that 
learning  is  improved  when  narrators  speak  in  a  conversational  style  (Mayer,  2003)  and  with  a 
standard  accent  rather  than  a  machine  voice  or  foreign  accent  (Mayer,  2003). 

The  authors’  experience  strongly  suggests  that  multiple  voices  should  be  used  for  narration. 
There  are  two  main  reasons  for  this  recommendation: 

1.  Use  of  multiple  voices  provides  variety  and  helps  maintain  the  learner’s  attention 

2.  When  revising  a  course,  it  is  not  likely  that  the  original  speaker  will  be  available  to 
change  the  narration.  Therefore,  a  new  voice  will  not  distract  the  learner  because  it  is 
one  of  many. 

In  terms  of  language  style,  the  narrator  should  speak  in  “plain  English.”  The  use  of  technical 
terms  should  only  be  used  when  necessary  to  support  the  content,  not  to  sound  “important.” 
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Jargon  should  never  be  use.  Furthermore,  the  narrator  should  never  talk  “at  the  learner,”  rather 
he/she  should  talk  “to  the  learner.” 

One  of  the  main  goals  of  narration  should  be  to  cognitively  engage  the  learner  with  the 
content  (e.g.,  make  them  think  about  it).  Therefore,  the  narration  should  include  important 
questions  about  the  content  and  attempt  to  draw  the  learner  into  deeper  understanding  of  the 
material. 

Graphics.  Bostock  (2004)  states  that  images  used  in  instructional  products  should  always  be 
unambiguous.  That  is,  they  should  directly  support  the  instructional  objectives  and  be  visually 
sharp  enough  that  the  learner  can  see  all  the  necessary  information  in  the  graphic. 

Levie  and  Lentz  (1982)  discuss  four  functions  of  graphics: 

•  Attentional :  graphics  can  attract  attention  to  important  content  and  direct  attention  within 
the  content. 

•  Affective:  graphics  can  enhance  enjoyment  and  stimulate  emotions  and  attitudes. 

•  Cognitive :  graphics  can  be  used  to  increase  comprehension  (i.e.,  providing  elaboration 
for  textual  explanations),  improve  retention,  improve  recollection,  and  provide 
information  that  is  not  otherwise  available. 

•  Compensatory:  graphics  can  assist  poor  readers  by  adding  pictorial  clues  to  decode  text. 

Lee  and  Boling  (1999)  provide  the  following  guidelines  for  use  of  graphical  images  (p.  21): 

•  Use  simple,  clear  images.  Images  with  too  much  detail  can  be  lost  when  presented  at  a 
small  scale  like  a  computer  screen. 

•  Only  use  graphics  for  instructional,  motivational,  or  attention-focusing  effects.  Never 
include  gratuitous  graphical  images. 

•  Make  sure  all  of  the  key  components  of  the  graphical  image  are  labeled.  Use  captions  or 
titles  for  labeling  the  key  elements. 

•  Avoid  sexist,  culturally-insensitive,  or  other  potentially  offensive  imagery. 

•  Obey  any  existing  conventions  (e.g.,  standard  symbols  in  a  circuit  diagram  or  top-to- 
bottom  or  left-to-right  order  for  a  flow  chart). 

Levie  and  Lentz  (1982)  reviewed  46  experimental  comparisons  of  reading  with  and  without 
pictures.  They  found  that  the  presence  of  relevant  illustrations  significantly  helped  the  learning 
of  text  information  in  85%  of  the  cases.  On  the  average,  groups  reading  with  relevant  pictures 
learned  1/3  more  than  groups  reading  text  alone. 

Anglin,  Towers,  and  Levie  (1996)  reviewed  a  large  body  of  literature  on  the  learning  effects 
of  static  and  dynamic  illustrations.  They  concluded  that  “static  visual  illustrations  can  facilitate 
the  acquisition  of  knowledge  when  they  are  presented  with  text  materials”  (p.  766).  They  further 
concluded  that  the  instructional  effectiveness  of  dynamic  illustrations  (e.g.,  animations)  is 
positive  when  motion  is  a  critical  attribute  of  the  concept  being  presented  and  motion  is  used  to 
cue  or  draw  the  learner’s  attention  to  the  material  being  presented. 
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Mayer,  Bove,  Bryman,  Mars,  and  Tapangco  (1996)  conducted  three  separate  experiments 
comparing  lessons  in  which  detailed  text  explained  the  visuals  to  lessons  that  only  included 
summary  text  and  visuals.  Learners  who  read  the  summary  text  performed  better  on  tests  of 
retention  and  transfer  than  those  who  read  the  detailed  passages.  The  graphics  appeared  to 
positively  supplement  the  summary  text. 

Color  and  Unused  Space.  Use  of  colors  can  enhance  the  impact  and  readability  of  text  by 
highlighting  important  words  and  definitions  (Bostock,  2004).  The  inclusion  of  white  or  unused 
space  can  also  reduce  clutter  and  enhance  readability.  Grabbinger  (1993)  found  that  multimedia 
screens  that  used  white  space  were  rated  higher  by  viewers  than  more  crowed  screens. 

In  general,  background  colors  or  patterns  should  not  detract  from  or  obscure  text  and 
graphics.  Hall  and  Hanna  (2004)  examined  the  impact  of  text-background  color  on  readability 
and  retention  of  information  on  web  pages.  Participants  viewed  either  an  educational  or 
commercial  web  page  for  ten  minutes.  Then,  they  completed  a  quiz/questionnaire  about  the 
content  of  the  web  sites.  The  participants  were  assigned  to  one  of  four  font/background  color 
conditions:  black/white,  white/black,  light  blue/dark  blue,  and  cyan/black.  Results  generally 
showed  that  the  pages  with  high  contrast  color  combinations  (B/W  and  W/B)  lead  to  greater 
readability.  Ratings  of  readability  were  positively  related  to  retention  for  the  educational  web 
site,  but  not  for  the  commercial  web  site.  Participants  indicated  that  they  preferred  the  blue  color 
schemes  on  the  commercial  site.  This  preference  was  positively  related  to  their  intention  to 
purchase  a  product.  These  results  highlight  the  importance  of  readability  for  instructional 
applications.  They  also  demonstrate  the  differences  between  commercial  and  instructional  web 
sites  in  terms  of  their  intended  purpose. 

Screens  should  be  free  from  flickering,  moving,  or  blinking  words  or  graphics  except  for 
specific  functions  to  gain  attention.  When  using  color  to  cue  the  learner,  Tognazzini  (2003) 
recommends  that  clear,  secondary  cues  (e.g.,  boldface)  should  also  be  provided  to  convey 
information  to  learners  who  may  be  color  blind. 

The  following  are  Lee  and  Boling’s  (1999)  guidelines  for  the  use  of  color  (pp.  21-22): 

•  Use  color  in  a  conservative  way  (e.g.,  limit  the  number  and  amount  of  colors  used  to  a 
maximum  of  three  to  seven  colors  per  screen). 

•  Keep  color  coding  consistent  (e.g.,  use  the  same  colors  for  menus,  titles,  buttons,  etc.). 

•  Use  colors  selectively  to  manipulate  attention  (e.g.,  a  bright  color  to  cue  the  learner  to 
new  information). 

•  Use  cool,  dark,  low-saturation  colors  (e.g.,  olive  green,  gray,  blue,  brown,  dark  purple, 
black,  etc.)  for  backgrounds.  Use  hotter,  lighter,  and  more  highly-saturated  colors  (e.g., 
lemon  yellow,  pink,  orange,  red,  etc.)  for  foregrounds. 

•  Avoid  the  use  of  complementary  colors  (e.g.,  blue/orange,  red/green,  violet/yellow). 

•  Use  commonly  accepted  colors  for  particular  actions  (e.g.,  red  for  stop,  green  for  go). 
However,  remember  that  such  colors  may  be  appropriate  only  for  a  specific  culture  or 
social  system. 

•  Use  higher  levels  of  brightness  for  distinguishing  colors  when  designing  for  younger 
audiences. 
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Animation  and  3-D  Graphics. 

Based  on  their  review  of  the  literature  on  teaching  equipment  procedures,  Hamel  and  Ryan- 
Jones  (1997)  concluded  that  “Emerging  technologies  such  as  3-D  graphics  and  animation  create 
new  visualizations  of  equipment  and  support  more  natural  interaction  with  the  computer”  (p.  85). 
They  cautioned,  however,  that  these  approaches  “should  be  applied  selectively  based  on  the 
cognitive  requirements  of  procedural  tasks”  (p.  86). 

Hamel  and  Ryan- Jones  (1997)  provided  several  guidelines  for  using  animations  and  3-D 
graphics.  Some  of  these  guidelines  include: 

•  Allow  learners  to  have  control  over  views  of  a  3-D  object  and  additional  textual 
elaboration.  Guidance  on  effective  viewing  strategies  should  be  provided. 

•  The  learner’s  attention  should  be  directed  to  the  important  parts  of  the  animated 
instruction. 

•  When  using  3-D  objects,  provide  aids  to  enhance  the  interpretation  of  the  object.  For 
example,  one  might  provide  additional  2-D  views  of  an  image  to  help  the  learner 
interpret  the  3-D  view. 

•  When  rotating  objects,  rotate  at  a  slow  rate  (about  equal  to  the  viewer’s  rate  of  mental 
rotation).  Sollenberger  and  Milgram  (1993)  found  support  for  this  guideline.  They 
found  that  visualizations  of  a  tree  structure  of  blood  vessels  in  the  brain  was  better  at 
slower  rotational  rates  as  long  as  the  rate  was  not  so  slow  as  to  remove  the  apparent 
motion  effect. 

•  Use  animations  as  a  visual  analogy  or  cognitive  anchor  for  the  instruction  of  problem 
solving  (see  also  Mayer  &  Anderson,  1992). 

•  Include  practice  and  feedback  in  animated  instruction  when  feasible.  For  example,  an 
exercise,  based  on  an  animation  could  use  a  version  of  the  animation  to  highlight  errors 
and  direct  the  learner  to  correct  the  procedure. 


Video.  The  use  of  video  can  provide  useful  support  for  many  instructional  approaches, 
including: 

•  “Talking  Heads”  for  lectures  or  expert  testimonials 

•  “Show  and  Tell”  for  demonstrations 

•  Panel  discussions 

•  Simulations  (especially  of  processes) 

•  “Walk  Throughs”  of  facilities  or  events 

Thibodeau  (1997)  provides  the  following  recommendations  when  using  video: 

•  Ensure  the  video  is  relevant  to  the  content  (e.g.,  only  if  content  requires  showing 
movement) 

•  Avoid  sensory  (modality)  conflicts  (see  Criterion  6.a) 

•  Show  the  consequences  of  improper  performance 

•  Make  sure  the  area  of  focus  is  well  lighted 
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•  Make  sure  there  are  no  distracting  movements,  such  as  someone  walking  by  in  the 
background  (the  learner’s  eye  will  be  attracted  to  the  movement) 

•  Use  long,  medium,  and  then  close-up  shots  to  establish  the  visual  introduction 

•  Use  close-ups  to  get  the  learner’s  attention  and  indicate  that  the  video  is  important 

•  Don’t  use  video  for  static  shots  (use  photographs) 

•  Zoom-in  is  useful  to  focus  learners’  attention  on  a  specific  object  or  area  while  still 
maintaining  their  visual  bearings 

•  When  introducing  something  new,  be  sure  to  focus  on  the  object  or  area  long  enough  to 
“register”  with  the  learner 


6.c.  Screens  are  Aesthetically  Pleasing 

Learners  are  more  likely  to  attend  to  and  be  motivated  by  instructional  presentations  that  are 
aesthetically  pleasing.  Although  an  individual’s  aesthetics  preferences  are,  in  some  part, 
subjective,  there  are  some  general  principles  that  can  enhance  the  aesthetics  of  media 
presentations. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  many  of  the  screens  in  the  course  are  aesthetically 
pleasing.  Use  the  anchor  statements  and  examples  of  aesthetic  principles  that  are  found  below 
the  scale  to  help  assign  a  rating  on  this  criterion.  Additional  information  is  provided  in  the 
discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

An  aesthetic  experience  is  one  which  has  “a  satisfying  emotional  quality  because  it  possesses 
internal  integration  and  fulfillment  reached  through  ordered  and  organized  movement”  (Dewey, 
quoted  in  Misanchuk,  Schwier,  &  Boling,  2000,  p.  2  of  10).  Misanchuk  and  colleagues  (2000) 
observe  that  aesthetics  are  not  a  quality  of  objects.  They  are  rather  found  in  one’s  experience 
when  viewing  or  working  with  an  object.  Nevertheless,  artists  have  worked  for  centuries  to 
develop  techniques  that  increase  the  aesthetic  appeal  of  their  work.  Many  of  these  techniques 
can  be  used  to  increase  the  aesthetic  appeal  of  instructional  content  and  the  learners’  engagement 
with  the  content  (Graham,  et  al.,  2000).  Likewise,  the  study  of  human  perception  has  helped  our 
understanding  of  how  to  present  instruction  that  supports  rather  than  conflicts  with  the  ways  that 
learners  perceive. 

Screen  Complexity.  Coll  and  Wingertsman  (1990)  found  that  the  relationship  between  screen 
complexity  and  user  performance  and  preference  follows  an  inverted  U-shaped  curve.  In  their 
first  experiment,  participants  were  assigned  to  one  of  three  screen  density  conditions  in  which 
they  had  to  complete  the  same  task.  Screen  density  referred  to  the  amount  of  information 
presented  on  a  screen  and  the  number  of  operations  available  to  the  user  on  the  screen.  Three 
levels  of  screen  density  were  used  in  the  study  (24%,  37%,  and  64%).  The  highest  density 
screen  contained  more  information  and  allowed  the  users  to  complete  more  operations  from  that 
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initial  screen.  Alternatively,  the  low  density  screens  contained  less  information  and  required  the 
users  to  navigate  through  multiple  screens  to  access  the  same  information  provided  on  the  first 
page  of  the  high  density  screen.  The  time  for  task  completion  was  fastest  and  appeal  scores  were 
highest  in  the  37%  condition.  In  either  the  lowest  or  highest  complexity  conditions,  they  found 
that  performance  times  were  longer  and  appeal  scores  were  lower. 

In  their  second  experiment,  they  asked  participants  which  screen  density  they  would  prefer  in 
a  learning  application.  Sixty  percent  of  the  subjects  chose  the  medium-complexity  (37%)  screen. 
The  authors  concluded  that  medium  complexity  screen  designs  should  result  in  both  better 
performance  and  higher  preferences  from  users. 

Screen  Design  Principles.  Skaalid  (1999)  and  Misanchuk,  et  al.  (2000)  have  compiled  screen 
design  guidelines  for  multimedia  and  web-based  instruction.  These  guidelines  are  based  on: 

•  “Classic  graphic  design  theory.”  These  include  elements  of  graphic  design  (e.g.,  use  of 
lines,  shapes,  textures,  colors,  and  shapes)  and  principles  of  graphic  design  (e.g., 
movement,  balance,  emphasis,  and  unity). 

•  Gestalt  principles  of  perception  (e.g.,  figure  and  ground,  similarity,  proximity, 
continuity,  closure,  area,  and  symmetry). 

•  Human-computer  interface  design  principles  (e.g.,  from  sources  such  as  Norman,  1988; 
Nielsen,  1994;  and  Schneiderman,  1998). 

There  is  not  sufficient  space  in  this  report  to  provide  details  on  all  of  these  design  topics.  It  is 
strongly  recommended  that  Instructional  Developers  and  Managers  ensure  that  their 
development  teams  have  copies  of  these  guidelines.  This  will  enable  the  teams  to  access  specific 
guidance  to  support  the  design  needs  of  their  instructional  products.  Below  are  a  few  brief 
samples  of  guidance  from  each  of  the  three  areas.  It  should  be  remembered  that  the  three  areas 
are  not  separate,  but  strongly  interact  with  and  influence  one  another. 

Classic  Graphic  Design  Theory.  One  of  the  most  important  aspects  of  aesthetically  pleasing 
instructional  design  is  simplicity.  Research  reported  by  Reeves  and  Nass  (1996)  led  them  to 
conclude  that  “People  like  simplicity....  Rather  than  empowering  through  endless  choices,  media 
can  empower  through  ease  of  use,  and  that  often  means  freedom  from  choice....  ‘Simple’  also 
means  predictable....  When  people  know  what  to  expect,  they  can  process  media  with  a  greater 
sense  of  accomplishment  and  enjoyment”  (p.  254).  Misanchuk  et  al.  (2000)  recommend  that  the 
instructional  message  be  presented  as  simply  as  possible,  without  any  confounding  or 
superfluous  elements  that  can  distract  the  learner.  For  example,  complex  or  “busy”  backgrounds 
should  be  avoided  and  diagrams  or  pictures  should  provide  just  enough  visual  information  to 
support  the  instruction.  Too  much  information  can  become  “noise,”  distracting  the  learner  from 
the  important  portions  of  the  instruction. 

Another  important  aspect  of  aesthetically  pleasing  instructional  design  is  the  use  of  color. 
Misanchuk  et  al.  (2000)  provided  several  reasons  to  use  or  not  to  use  color  (color,  pros  and  cons, 
p.  2  of  3).  Some  of  the  reasons  to  use  color  include: 

•  Color  may  be  necessary  to  convey  important  instructional  information  (e.g.,  how  a 
diseased  organ  looks  compared  to  a  normal  one). 
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•  Color  may  help  learners  to  organize  instructional  content  and  help  them  process  and 
retrieve  information. 

•  Learners  like  color. 

Some  of  the  reasons  not  to  use  color  include: 

•  Many  instructional  materials,  especially  text  on  a  screen,  do  not  require  color. 

•  Color  can  be  a  distraction  to  the  learner. 

•  Not  all  learners  have  a  computer  that  will  present  the  colors  correctly  (monochrome  is  the 
lowest  common  denominator). 

•  Some  learners  may  be  colorblind  or  have  difficulty  distinguishing  certain  color 
differences. 

Principles  of  Perception.  Two  perceptual  principles  that  are  well  supported  by  research  are 
proximity  and  similarity.  “The  Principle  of  Proximity  suggests  that  objects  that  are  close 
together  are  interpreted  as  being  related,  while  objects  that  are  apart  are  viewed  as  being 
unrelated”  (Misanchuk,  et  al.,  2000,  perceptual  principles,  p.  2  of  24).  Thus,  locating  related 
objects  close  together  on  a  computer  screen  provides  cues  to  help  the  learner  perceive  this 
relationship.  “The  Principle  of  similarity  states  that  objects  that  look  alike  are  perceived  as 
belonging  together”  (p.  6  of  24).  Similarity  is  so  powerful  that  it  can  sometimes  override 
proximity.  These  two  principles  should  be  used  in  tandem  to  help  the  learner  perceive  patterns 
in  the  instructional  content.  These  patterns  help  in  both  learning  and  retrieval  of  the  content. 

Human-Computer  Interface  Design  Principles.  Over  the  years,  researchers  have  learned 
many  things  about  how  people  interact  with  computers.  This  knowledge  has  been  accumulated 
into  computer  interface  design  principles.  One  of  the  most  widely  used  set  of  principles  is  the 
usability  heuristics  recommended  by  Molich  and  Nielsen  (1990)  and  Nielsen  (1993).  Many  of 
these  were  listed  in  the  Introduction  to  Section  III.  Although  these  heuristics  are  focused  on  any 
type  of  computer  interface,  they  can  easily  be  adapted  to  meet  the  interface  design  requirements 
of  computer-delivered  instruction.  Applications  of  these  heuristics  are  also  discussed  under 
other  criteria  (especially  Criteria  5.e,  5.f,  and  5.g). 


6.d.  Multi-modal  Presentation  of  Content  is  Used 


Computer-delivered  instruction  can  present  instructional  material  that  is  focused  on  different 
senses  or  combinations  of  senses  (e.g.,  visual,  auditory).  Instruction  that  uses  various  senses  is 
sometimes  referred  to  as  multi-modal. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  often  different  presentation  modes  are  used  to  reinforce 
and  clarify  instructional  information.  Use  the  anchor  statements  and  the  list  of  examples  of 
multi-modal  presentation  found  below  the  scale  to  help  assign  a  rating  on  this  criterion. 
Additional  information  is  provided  in  the  discussion  below. 
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Terms.  Concepts.  Techniques,  and  Supporting  Literature 

Some  research  (e.g.,  Moreno  and  Mayer,  1999)  shows  that  learners  benefit  when  instruction 
is  presented  using  multiple  sensory  modalities  (e.g.,  vision,  sight,  hearing).  However,  as 
mentioned  above  (6.a),  there  are  large  risks  of  creating  sensory  conflict  if  the  information  is 
different  across  the  modalities. 

Sharon  Oviatt  (1999)  discusses  “myths”  about  multimodal  interaction  that  she  terms 
“currently  fashionable  among  computationalists”  (p.  75).  These  “myths”  are  all  refuted  to  some 
degree  by  empirical  data.  She  discusses  most  of  the  “myths”  in  the  context  of  operational 
systems,  but  nonetheless,  some  have  some  relevance  for  multi-modal  presentation  of 
instructional  content.  For  example: 

•  Myth:  If  you  build  a  multi-modal  system,  users  will  interact  multi-modally .  Even  though 
they  like  to  interact  multi-modally,  users  do  not  always  do  so.  They  often  use  a  mix  of 
uni-  and  multi-modal  commands,  depending  on  the  nature  of  the  action.  Instructional 
interface  designers  should  carefully  determine  the  input  options  they  provide  for  their 
course.  Standard,  point  and  click  inputs  are  probably  sufficient. 

•  Myth:  Multi-modal  input  involves  simultaneous  signals.  Users  of  operational  systems 
often  synchronize  their  multi-modal  inputs,  but  synchrony  does  not  imply  simultaneity. 
Simultaneous  presentation  of  instructional  content  risks  overloading  the  learner  with 
information.  Designers  should  use  different  modalities  to  reinforce  important  items  of 
information  and  help  tie  them  together.  The  specific  timing  of  each  modality  needs  to  be 
determined  by  the  nature  of  the  information  to  be  presented. 

•  Myth:  Multi-modal  integration  involves  redundancy  of  content  between  modes.  “The 
dominant  theme  in  user’s  natural  organization  of  input  actually  is  complementarity  of 
content,  not  redundancy”  (Oviatt,  1999,  p.  78).  Instructional  designers  should  search  for 
a  balance  between  redundancy  (repetition  to  aid  deeper  learning)  and  complementarity 
(helping  learners  link  pieces  of  instructional  information  or  procedural  steps  together). 

•  Myth:  All  users  ’  multi-modal  commands  are  integrated  in  a  uniform  way.  Individual 
differences  in  the  way  individuals  integrate  multi-modal  interactions  require  careful 
choices  in  modes  and  mixtures  of  modes  for  content  presentation. 

•  Myth:  Different  input  modes  are  capable  of  transmitting  comparable  content.  “Different 
modes  basically  vary  in  the  degree  to  which  they  are  capable  of  transmitting  similar 
information  with  some  modes  relatively  more  comparable  (speech  and  writing)  and 
others  less  so  (speech  and  gaze). 


6.e.  Multi-media  Presentation  of  Content  is  Used 


Computer-delivered  instruction  affords  many  presentation  options.  These  presentation  options 
include:  text,  graphics,  video,  animations,  and  others.  Collectively,  these  presentation  options 
are  often  referred  to  as  multi-media. 
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Using  the  Scale 

This  scale  is  used  to  evaluate  how  many  different  media  are  used  in  the  course.  The  anchor 
statements  and  list  of  example  instructional  media,  found  below  the  scale,  should  be  used  to 
assist  in  assigning  a  rating  on  this  criterion.  Additional  information  is  presented  in  the 
discussion  below. 

Terms,  Concepts.  Techniques,  and  Supporting  Literature 

Multi-media  is  “a  computer-driven  interactive  communication  system  for  various  types  of 
textual,  graphic,  video,  and  auditory  information”  (Tessmer,  1995,  p.  128;  see  also  Gayeski, 
1992,  p.  9).  A  multi-media  presentation  can  be  instructionally  beneficial  if  it  enhances  the 
content  and  makes  it  easier  to  understand  and  remember.  Media  should  never  be  included 
gratuitously  because  they  can  become  distractions.  Furthermore,  as  noted  in  Criteria  6.a  and  6.d, 
designers  should  ensure  that  media  do  not  present  information  in  conflicting  sensory  modalities. 

Media  integration  refers  to  how  well  different  media  are  combined  to  produce  an  effective 
whole  or  how  they  work  together  to  form  one  cohesive  program.  Jones  and  Okey  (1995) 
suggested  that  media  be  should  be  designed  so  that  learners  can  search  for  information  across 
different  media.  They  also  observed  that  some  types  of  media  may  be  perceived  as  more 
credible  than  others  (e.g.,  text  over  video). 

A  number  of  researchers  have  reviewed  research  on  multi-media  effects  on  learning  (e.g., 
Mayer,  2001).  Below  are  some  instructional  design  principles  that  have  resulted  from  these 
efforts. 

•  Media  Choice  Principle:  Choose  the  medium  that  best  communicates  the  information 
to  be  learned  (Najjar,  1998).  For  example: 

o  For  a  small  amount  of  information  to  be  remembered  for  a  short  time,  audio  is 
better  than  text. 

o  Text  is  better  than  sound  for  longer  retention, 
o  Pictures  are  useful  for  presenting  spatial  information, 
o  Simple  illustrations  with  captions  are  more  effective  than  text  for  summarizing 
information  (Mayer,  Bove,  Bryman,  Mars,  &  Tapangco,  1996). 

•  Multimedia  Principle:  Learners  learn  better  from  words  and  pictures  than  from  words 
alone  (Mayer,  2003,  p.  37;  Mayer  &  Anderson,  1991). 

•  Spatial  Contiguity  Principle:  Learners  learn  better  when  corresponding  words  and 
pictures  are  presented  near  rather  than  far  from  each  other  on  the  page  or  screen  (Mayer, 
2003,  p.  49). 

•  Temporal  Contiguity  Principle:  Learners  leam  better  when  corresponding  words  and 
pictures  are  presented  so  they  coincide  meaningfully  (Mayer,  2003,  p.  51;  Mayer  & 
Anderson,  1992). 

•  Irrelevancy  Principle:  Learners  leam  better  when  extraneous  words,  pictures,  and 
sounds  are  excluded  rather  than  included  (Mayer,  2003,  p.  33;  Harp  &  Mayer,  1998; 
Moreno  &  Mayer,  2000a).  Gratuitous  use  of  media  can  detract  from  learning.  As  Levie 
and  Dickie  (1973)  put  it,  “Realism  cues  which  are  not  relevant  to  the  learning  task  may 
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produce  divergent  responses  especially  under  conditions  of  high  information  loads”  (p. 
875).  Unrelated  illustrations  “may  actually  decrease  learning”  (Najjar,  1998,  p.  313). 

•  Modality  Principle:  Learners  learn  better  from  animation  and  narration  than  from 
animation  and  on-screen  text  (Mayer,  2003,  p.  35).  This  has  been  called  the  “split 
attention  effect”  (Mayer  &  Moreno,  1998;  Mousavi,  Low,  &  Sweller,  1995).  However, 
this  general  principle  must  be  tempered  by  specific  guidance  on  the  type  of  material  to  be 
learned  and  the  conditions  in  which  a  given  modality  is  applied.  For  example,  Rieber 
(1990)  found  that  fourth-  and  fifth-grade  students  learned  science  principles  better  from 
animated  presentations  only  if  practice  was  provided.  Thus,  the  specific  conditions  that 
improve  learning  of  specific  material  must  be  understood  before  decisions  are  made  to 
use  one  modality  over  another.  Another  important  issue  when  delivering  instructional 
information  using  multiple  modalities  is  the  timing  of  delivery.  Kalyuga  (2000)  found 
that  combining  a  diagram,  text,  and  narration  can  result  in  cognitive  overload  and 
reductions  in  learning  if  the  text  and  narration  are  presented  simultaneously.  When  it  is 
“necessary”  for  narration  to  be  redundant  with  text,  Kalyuga  recommends  that  written 
materials  be  delayed  and  presented  after  the  auditory  narration  has  been  fully  articulated. 
Another  technique  to  avoid  this  problem  is  to  present  text  as  “bullets”  rather  than  full 
text. 

•  Redundancy  Principle:  “Learning  is  facilitated  by  increasing  the  redundancy  of 
relevant  cues  and  reducing  the  number  of  cues  that  are  irrelevant  to  the  learning  task” 
(Levie  and  Dickie,  1973,  p.  875;  Mayer,  2003,  p.  45).  Media  should  elaborate  the 
instructional  information  (Najjar,  1998,  p.  313;  Kalyuga,  Chandler,  &  Sweller,  1999; 
Clark,  R.  C.,  &  Mayer,  2003;  Moreno  &  Mayer,  2002),  but  should  not  include  features 
that  are  interesting  but  not  relevant  (Harp  &  Mayer,  1997;  Mayer,  Heiser,  &  Lonn,  2001). 
Fox  (2004)  found  support  for  this  principle.  Using  signal  detection  methods,  she  found 
that  subjects  were  better  able  to  discriminate  information  from  video  news  stories  with 
redundant  visuals  than  from  news  stories  with  dissonant  visuals. 

•  Interaction  Principle:  A  cognitively  engaging,  interactive  user  interface  “appears  to 
have  a  significant  positive  effect  on  learning  from  multimedia”  (Najjar,  1998,  p.  314). 

•  Prior  Knowledge  Principle:  Media  design  effects  are  stronger  for  learners  with  low- 
knowledge  of  subject  area  and  lower  aptitudes  (Mayer,  &  Gallini,  1990).  High 
knowledge  and  high  aptitude  learners  seem  to  benefit  from  almost  any  media  design. 
(Mayer,  2003,  p.  43). 

•  Voice  Principle:  People  learn  better  from  narration  when  the  voice  is  human  (rather 
than  a  machine  voice)  and  speaks  with  a  standard  accent  (Mayer,  2003,  p.  53). 

•  Personalization  Principle:  Learning  is  facilitated  in  multimedia  lessons  when  the  words 
are  in  conversational  style  rather  than  formal  style  (Mayer,  2003,  p.  39). 

•  Pretraining  Principle:  People  learn  better  from  multimedia  when  they  already  know 
something  about  the  topic  (e.g.,  names  and  functions  of  components)  (Mayer,  2003,  p. 

41;  Mayer,  Mathias,  &  Wetzell,  2002). 

•  Signaling  Principle:  Multimedia  explanations  using  narrated  animations  should  include 
highlights  of  the  key  steps,  section  headings  that  correspond  to  the  key  steps,  and/or  other 
techniques  to  signal  the  importance  of  the  information  (Mayer,  2003,  p.  47). 
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•  Pacing  Control  Principle:  The  cognitive  load  imposed  on  the  learner  can  be  reduced 
and  deeper  learning  can  be  achieved  by  allowing  the  learner  to  control  the  rate  of 
presentation  (Mayer  &  Chandler,  2001). 

•  Appropriate  Instructional  Cues  Principle:  The  instructional  medium  or  mix  of  media 
should  be  chosen  on  the  basis  of  the  media  attributes  that  will  facilitate  the  learning  of 
specific  tasks.  Levie  and  Dickie  (1973)  summarize  the  need  for  additional  research  on 
which  specific  media  attributes  should  be  used  for  instruction  on  specific  tasks.  “No 
single  level  of  independent  variable  is  consistently  superior”  (p.  877).  They  recommend 
that  instructional  researchers  should  direct  their  efforts  toward  “discovering  the 
conditions  under  which  different  levels  of  attributes  are  differentially  effective.  What 
media  attributes  will  facilitate  learning  for  what  kinds  of  tasks?”  (p.  877). 

6.f.  Media  are  Easy  to  Use 


If  instructional  media  are  not  easy  to  use,  it  may  lead  to  frustration,  distraction,  and  loss  of 
learner  motivation. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  often  the  media  in  the  course  are  easy  to  use.  The  anchor 
statements  and  list  of  examples  of  easy-to-use  media,  found  below  the  scale,  should  be  consulted 
to  help  assign  a  rating  on  this  criterion.  Additional  information  is  provided  in  the  discussion 
below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

Learners  should  not  have  to  struggle  to  operate  an  instructional  delivery  medium.  Controls 
should  be  easily  located,  clearly  labeled,  and  easy  to  operate.  Media  will  be  easy  to  use  if 
designers  follow  the  requirements  of  Criteria  5.d,  5.e,  5.f,  and  5.g.  One  example  of  a  difficult-to- 
use  medium  would  be  if  the  control  for  a  video  presentation  were  hidden  by  another  window. 


6,g,  External  Hyperlinks  are  Kept  to  a  Minimum 

The  authors  have  met  some  people  who  believe  that  one  of  the  strengths  of  web-delivered 
instruction  is  the  ability  for  learners  to  link  to  other  web  sites  for  additional  information. 
Including  hyperlinks  to  other  web  sites  can  increase  the  learner’s  frustration  if  these  links  do  not 
function  properly  or  if  the  sites  have  been  modified  since  the  course  was  developed.  It  is  the 
position  of  this  criterion  that  external  hyperlinks  should  be  few  or  not  used  at  all. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  many  external  hyperlinks  are  used  in  the  course.  It  is 
assumed  that  fewer  external  hyperlinks  is  always  to  be  preferred.  The  anchor  statements  below 


137 


Technical  Report  2005-002 


the  scale  should  be  used  to  help  assign  a  rating.  Additional  information  is  provided  in  the 
discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

Informational  sites  are  usually  designed  to  sell  a  product  or  provide  information  chosen  by 
the  user.  On  the  other  hand,  instructional  sites  are  designed  to  provide  structured  instructional 
information  and  to  assess  whether  the  learner  has  mastered  this  information.  Acknowledging 
these  differences,  there  is  still  much  to  be  learned  from  commercial  web  sites. 

Many  links  used  on  web  sites  fail  to  make  clear  their  destinations.  Borges,  Morales,  and 
Rodriguez  (1998)  investigated  users’  ability  to  predict  the  destinations  of  50  links  randomly 
selected  from  10  commercial  web  sites.  They  found  that  “in  approximately  one  fourth  of  the 
cases,  the  link  names  suggested  a  wrong  idea  about  the  content  of  a  page”  (p.  145,  cited  in 
Farkas  &  Farkas,  2000,  p.  5  of  21).  “Even  in  well-designed  Web  sites,  users  will  periodically 
follow  a  link  to  an  unwanted  node  or  will  sit  and  ponder  whether  to  follow  a  link.  But  designers 
should  work  hard  to  minimize  these  frustrating  occurrences  (Farkas  &  Farkas,  2000,  p.  5  of  21). 

“Entropy  is  the  enemy  of  the  Web,  which  tends  toward  disorder.  Your  sites — and  the  outside 
Web  sites  you  link  to — get  dated,  changed,  or  reorganized.  Inevitably,  links  break.  In  their 
initial  enthusiasm,  many  new  Web  authors  add  dozens  of  external  links  to  their  sites. 
Unfortunately,  each  of  those  links  soon  becomes  a  maintenance  issue”  (Lynch  &  Horton,  1997, 
p.  117).  Links  to  websites  that  do  not  support  LOs  could  reflect  poorly  on  your  instruction. 
Furthermore,  if  the  links  are  not  properly  maintained,  they  could  provide  incorrect  or  irrelevant 
information.  In  any  case,  external  links  introduce  possible  problems  because  they  are  outside  of 
your  control. 
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Chapter  9: 

Installation  and  Registration 
(Evaluation  Area  7) 

The  following  criteria  refer  to  any  special  actions  that  the  learner  needs  to  complete  to  begin  a 
course  and  receive  credit  for  successful  completion.  In  all  cases,  care  should  be  taken  to  make 
these  actions  as  easy  as  possible. 


7.a.  Course  Does  not  Require  Installation  or  Learners  Can  Install  the  Course 
without  Assistance 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  difficult  it  is  to  install  a  course  on  the  learner’s  computer. 
Use  the  anchor  statements  below  the  scale  to  help  assign  a  rating  on  this  criterion.  Additional 
information  is  provided  in  the  discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

Some  computer-delivered  courses  must  be  downloaded  from  an  external  source  (e.g.,  the 
internet)  and  installed  on  the  learner’s  computer.  Others  are  delivered  to  the  learner  on  an 
electronic  medium  (e.g.,  CD  ROM)  and  then  have  to  be  installed  on  his  or  her  computer.  Other 
courses  do  not  require  installation. 

When  installation  is  required,  as  it  often  is  to  ensure  the  learner  obtains  credit  for  the  course, 
it  should  be  easily  accomplished.  Easy  installation  will  minimize  distractions  for  the  learner  and 
also  avoid  demotivating  the  learner  by  putting  him  or  her  through  excessive,  frustrating  actions. 
The  installation  process  should  be  fully  explained  and  be  completed  in  a  short  time.  It  is 
important  to  get  the  learner  to  the  instructional  content  as  quickly  as  possible. 

Often  the  learner  may  start  a  course  from  one  location  (e.g.,  work)  and  wish  to  continue  or 
complete  it  at  another  location  (e.g.,  home).  If  this  is  the  case,  he  or  she  should  be  able  to  restart 
the  course  from  different  computers  without  losing  his  or  her  place  in  the  course  or  having  to 
retake  completed  lessons. 
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7.b  Minimal  “Plug-ins”  are  Required 


Using  the  Scale 

This  scale  is  used  to  evaluate  how  many  plug-ins  (extra  software  programs)  are  required  to 
run  the  course.  Use  the  anchor  statements  found  below  the  scale  to  assist  in  making  your  rating 
on  this  criterion.  Additional  information  is  provided  in  the  discussion  below. 

Terms,  Concepts.  Techniques,  and  Supporting  Literature 

“Plug-ins”  are  software  programs  that  are  needed  to  perform  functions  within  the  course.  An 
example  is  a  program  to  play  video  clips.  In  general,  the  course  should  be  self-contained.  All  of 
the  necessary  plug-ins  should  be  included  with  the  course  or  the  learner  should  be  provided  with 
easy  instructions  to  download  the  plug-in.  These  instructions  should  be  easy  from  the 
perspective  of  the  learner,  not  the  computer  programmer.  The  learner  should  not  be  required  to 
install  a  large  number  of  plug-ins,  nor  should  he  or  she  be  required  to  locate  the  plug-ins  prior  to 
installation. 

Many  plug-ins  are  included  in  standard  browser  software.  It  is  strongly  recommended  that 
developers  only  use  these  standard  plug-ins.  There  should  be  a  compelling  reason  to  use  a 
unique  plug-in.  For  example,  if  a  3-D  view  is  needed  to  demonstrate  a  procedure  and  only  a 
special-purpose  plug-in  will  work,  then  it  must  be  used.  However,  in  no  case  should  a  special- 
purpose  plug-in  be  used  because  it  is  easier  for  the  developer  or  because  the  developer  is  more 
familiar  with  it.  The  decision  to  use  a  plug-in  should  be  based  on  what  will  be  easier  for  the 
learner  and  what  will  support  the  learning  objectives  of  the  course. 


7.c,  “Optimization”  Test  is  Available 


Using  the  Scale 

This  scale  is  used  to  evaluate  how  well  the  optimization  test  provides  a  tryout  of  all  features 
and  provides  directions  to  set  up  the  computer  to  display  the  course  in  the  most  effective  manner. 
Use  the  anchor  statements  below  the  scale  to  assist  in  assigning  a  rating  on  this  criterion. 
Additional  information  is  provided  in  the  discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

Many  computers  differ  in  small  ways  that  can  change  the  appearance  or  other  functionality  of 
the  course.  The  course  instructions  or  tutorial  should  include  a  “test”  to  help  the  learner 
determine  if  his  or  her  computer  is  properly  set  up  for  the  course  (also  see  Criterion  5.b). 
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One  method  to  help  learners  determine  if  their  computer  is  configured  to  support  the  course  is 
to  include  several  test  functions.  This  could  be  a  series  of  pictures,  videos,  audio  tests,  and 
others.  The  learner  should  be  prompted  as  to  how  the  computer  should  be  functioning  (e.g.,  “can 
you  read  this  text?”  or  “can  you  see  the  red  dot?”).  If  the  learner’s  computer  is  not  functioning 
properly,  he  or  she  should  be  provided  with  detailed  instructions  to  properly  configure  the 
computer. 


7.d.  Technical  Support  is  Available 

Most  learners  are  not  computer  experts.  When  encountering  a  problem  with  the  course,  it  is 
important  that  they  have  access  to  technical  support.  This  technical  support  needs  to  get  them 
back  to  the  instructional  experience  as  soon  as  possible. 

Using  the  Scale 

This  scale  is  used  to  evaluate  how  much  technical  support  is  available  and  how  effective  it  is 
for  the  learner.  Use  the  statements  below  the  scale  to  help  assign  a  rating  on  this  criterion. 
Additional  information  is  provided  in  the  discussion  below. 

Terms,  Concepts,  Techniques,  and  Supporting  Literature 

Technical  support  needs  to  be  “learner-centric.”  This  means  that  the  needs  of  the  learner  are 
paramount  and  the  most  important  function  of  the  technical  support  staff  is  to  get  the  learner  into 
the  course  as  quickly  and  painlessly  as  possible. 


7,e.  Registration  is  Simple  and  Straightforward  (Or  Not  Required) 


Using  the  Scale 

This  scale  is  used  to  evaluate  how  easy  it  is  to  register  for  the  course  if  registration  is 
required.  Use  the  anchor  statements  below  the  scale  to  help  assign  a  rating  on  this  criterion. 
Additional  information  is  provided  in  the  discussion  below. 

Note:  If  no  registration  is  required  for  the  course,  assign  a  score  of  five  (5)  on  this  criterion. 

Terms.  Concents,  Techniques,  and  Supporting  Literature 

Learners  are  usually  required  to  register  for  a  course.  This  is  often  to  ensure  that  they  have 
completed  any  prerequisites  and  also  to  ensure  that  they  receive  credit  for  completing  the  course. 
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In  many  cases,  organizations  use  learning  management  systems  to  register  and  track  learners.  In 
other  cases,  the  course  may  include  software  that  records  and  transmits  registration  information. 
Either  way,  registration  should  be  as  easy  as  possible  for  the  learner.  He  or  she  should  be  able  to 
quickly  accomplish  the  task  and  move  on  to  the  course  itself.  Upon  completion,  he  or  she  should 
be  able  to  easily  receive  credit  for  the  course. 
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CHAPTER  10: 

SUMMARY  AND  RECOMMENDATIONS 


The  Instructional  Quality  Evaluation  Tool  provides  an  empirically-based,  quantitative 
method  to  evaluate  the  instructional  quality  of  computer-  and  web-delivered  instruction.  The 
anchored  evaluation  scales  cover  all  of  the  critical  areas  that  can  affect  the  quality  of  instruction. 
The  evaluation  scales  are  divided  into  two  major  sections:  Instructional  Features  and  User 
Interface.  Instructional  Features  are  evaluated  in  four  areas:  Instructional  Content,  Instructional 
Activities,  Performance  Assessment,  and  Performance  Feedback.  The  User  Interface  is 
evaluated  in  three  areas:  Navigation  and  Operation,  Content  Presentation,  and  Installation  and 
Registration.  Guidance  is  provided  on  the  interpretation  of  evaluation  scores  overall  and  in  each 
evaluation  area. 

Using  the  Tool 

Instructional  developers  and  instructional  program  managers  can  both  use  the  Tool  to  help 
ensure  the  highest  quality  in  their  instructional  products.  The  Tool  can  be  used  for  formative 
evaluations  during  product  development  and  also  for  summative  evaluations  when  the  product  is 
delivered.  During  formative  evaluations,  the  evaluation  criteria  can  help  focus  the  efforts  of 
instructional  developers  on  the  features  that  will  have  the  greatest  impact  on  the  quality  of  the 
product.  During  instructional  product  development,  the  evaluation  criteria  provide  a  “roadmap” 
for  program  managers  to  help  ensure  that  their  project  team  is  focused  on  the  issues  that  will 
optimize  instructional  quality.  The  evaluation  criteria  can  be  used  to  establish  discussion  points 
during  developmental  reviews  and  can  help  assign  resources  for  project  activities. 

During  summative  evaluations,  the  evaluation  scales  provide  an  empirically-based, 
quantitative  method  to  determine  if  the  instructional  product  has  met  its  instructional  goals.  If 
not,  the  program  manager  has  the  opportunity  to  revise  the  product  before  it  is  implemented  in  a 
formal  program  of  instruction. 

The  evaluation  criteria  can  also  help  instructional  program  managers  choose  among 
competing  instructional  product  development  companies.  When  evaluating  proposals,  the 
evaluation  criteria  can  be  used  to  differentiate  between  competitors  on  the  basis  of  their 
commitment  to  the  activities  that  will  result  in  higher  quality  instruction. 

Recommended  Developments 

This  report  presented  the  first  version  of  the  Instructional  Quality  Evaluation  Tool.  It  is 
recommended  that  the  Tool  undergo  additional  development  to  address  at  least  the  following 
improvements: 

•  Expand  discussion  sections.  The  discussion  sections  should  be  expanded  to  include 
additional  empirical  support  for  the  evaluation  criteria.  Any  gaps  in  the  empirical  support  for 
a  given  criterion  should  also  be  identified  to  provide  a  “road  map”  for  future  instructional 
research. 
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•  Conduct  reliability  and  validity  tests  of  the  criteria.  The  criteria  should  be  applied  to  a 
variety  of  instructional  programs  to  specifically  test  their  reliability  and  validity.  These  data 
will  help  future  evaluators  feel  more  confident  in  using  the  Tool. 

•  Conduct  formative  evaluations  of  the  Tool.  The  tool  should  be  used  by  several  individuals 
and  groups  of  evaluators  to  determine  the  most  effective  methods  to  conduct  evaluations. 
Currently  the  authors  believe  that  a  group  evaluating,  using  a  consensus  approach  is  the  most 
feasible.  However,  additional  research  may  not  support  this  assumption.  Also,  these 
formative  evaluations  can  provide  insights  into  how  to  apply  the  Tool  during  different  phases 
of  instructional  product  development.  Finally,  the  formative  evaluations  can  help  determine 
special  and  specific  guidance  that  may  be  needed  for  different  “types”  of  evaluators  (e.g., 
instructional  developers  versus  instructional  program  managers). 

•  Develop  weightings  for  criteria.  Not  all  evaluation  criteria  are  created  equal.  Some  may  be 
more  important  for  certain  types  of  content  or  for  certain  instructional  approaches.  Additional 
reviews  of  empirical  research  and  discussions  with  instructional  development  experts  can  help 
develop  a  weighting  scheme  for  the  criteria.  This  weighting  scheme  will  help  “tailor”  the 
evaluation  criteria  to  the  specific  needs  of  a  given  instructional  program. 

•  Develop  or  adapt  the  criteria  for  classroom  and  other  types  of  instruction.  The  focus  of  the 
current  Tool  is  on  computer-  and  web-delivered  instruction.  Other  types  of  instruction  should 
also  be  evaluated.  Adaptations  of  the  Tool  can  be  developed  to  aid  in  the  evaluations  of  these 
other  instructional  approaches. 

•  Examine  the  feasibility  of  computerized  scoring.  Evaluations  using  the  Tool  may  be 
facilitated  by  using  a  computerized  scoring  method.  This  computerized  scoring  system  would 
keep  track  of  the  scores  on  all  criteria,  sum  the  scores,  interpret  the  scores,  and  generate  an 
evaluation  summary.  If  weightings  are  developed  for  the  criteria,  this  system  could  also 
assign  these  ratings  based  on  a  few  questions  about  the  specific  nature  of  the  course  (e.g., 
knowledge  or  skill  based;  introductory  or  refresher,  etc.). 
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Appendix  A 

Recommendations  for  Teaching  Specific  Types  of  Content 

Teaching  Concepts 

Merrill  and  Tennyson  (1977)  provide  a  five  step  prescription  for  teaching  concepts.  Before 
beginning,  one  must  first  understand  what  is  involved  in  concept  learning.  A  concept  is  “a  set  of 
specific  objects,  symbols,  or  events  which  are  grouped  together  on  the  basis  of  shared 
characteristics  and  which  can  be  referenced  by  a  particular  name  or  symbol”  (Merrill  & 
Tennyson,  1977,  p.  12).  One  can  know  that  a  student  has  learned  a  concept  when  he  or  she  can 
correctly  classify  (identify  the  class  membership)  of  a  specific  symbol,  object,  or  event.  To 
successfully  classify  a  concept,  the  student  must  engage  in  two  psychological  processes: 
generalization  and  discrimination.  “Generalization  occurs  when  a  learner  exhibits  a  particular 
response  in  one  stimulus  situation  which  he/she  acquired  in  a  similar  stimulus  situation. 
Discrimination  occurs  when  a  learner  exhibits  a  particular  response  in  one  stimulus  situation  but 
a  different  response  in  a  similar  stimulus  situation”  (Merrill  &  Tennyson,  1977,  p.  13).  These 
should  not  be  confused  with  association  and  rule-using  behaviors.  “Association  behavior  occurs 
when  a  student  is  asked  to  associate  two  specific  symbols,  objects,  or  events  by  being  able  to 
name  one  when  shown  the  other.  Rule  using  behavior  occurs  when  a  student  is  asked  to 
combine  objects,  symbols,  or  events  from  two  or  more  classes  by  means  of  some  operation  to  for 
an  object,  symbol,  or  event  in  a  resulting  concept  class”  (p.  13). 


According  to  Merrill  and  Tennyson  (1977),  teaching  concepts  should  follow  these  steps 
(consult  the  original  for  details): 

1.  Decide  if  a  Concept  Lesson  is  Needed.  Ask  the  following  questions: 

•  Does  the  material  involve  new  terms?  If  the  answer  is  yes,  a  concept  lesson  is  needed 
for  each  new  term  or  related  set  of  new  terms.  Does  the  material  require  the  student 
to  define  new  words?  If  the  answer  is  yes,  a  concept  lesson  is  needed  for  each 
important  new  word  or  related  set  of  words. 

•  Does  the  content  involve  rule  using?  If  yes,  a  concept  lesson  is  needed  for  each 
component  concept  in  the  rule. 

•  Does  the  content  involve  a  series  of  steps  or  events?  If  yes,  examine  each  step  or 
event  as  a  potential  concept  and  prepare  a  concept  lesson  for  those  steps  or  events 
which  are  concepts. 

•  Does  the  material  require  identification  of  parts?  If  yes,  decide  which  parts  should  be 
taught  as  concepts  and  prepare  a  concept  lessons  for  those. 

2.  Define  the  Concept.  Follow  these  steps  to  define  the  concept: 

•  Identify  the  concept  name  (a  word  or  symbol  which  is  used  to  refer  to  the  class  as  a 
whole  or  to  examples  of  the  class). 

•  List  critical  and  variable  attributes  of  the  concept.  A  critical  attribute  is  a 
characteristic  necessary  for  determining  class  membership  of  a  given  instance.  A 
variable  attribute  is  a  characteristic  shared  by  some,  but  not  all,  members  of  the 
class. 
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Appendix  A 

Recommendations  for  Teaching  Specific  Types  of  Content 

(Continued) 

•  Write  a  concise  definition  of  the  concept  (a  statement  identifying  each  of  the  critical 
attributes  and  how  they  are  combined).  Definitions  can  specify  conjunctive, 
disjunctive,  relational,  or  combination  relationships  between  attributes. 

3.  Collect  an  Instance  Pool  of  Divergent  Examples  Each  Matched  to  a  Nonexample.  Follow 
these  steps: 

•  Decide  on  the  form  of  concept  representation  (examples)  to  be  used.  Indicate  the 
variety  needed  and  eliminate  any  forms  which  do  not  present  the  critical  attributes. 

•  The  instance  pool  should  be  as  diverse  as  possible. 

•  Each  example  should  be  matched  to  a  nonexample. 

4.  Estimate  the  Difficulty  for  Each  Instance.  Some  symbols,  objects,  or  events  in  a  concept 
class  will  be  more  difficult  to  classify  than  others.  Instance  difficulty  can  be  estimated  by 
determining  the  probability  of  a  given  instance  being  classified  correctly  by  a  sample  of 
students  who  have  been  given  only  a  definition. 

•  Present  the  instances  with  their  definitions  to  a  portion  of  the  target  population. 

•  Record  the  number  of  correct  and  incorrect  responses  for  each  instance  and 
determine  the  percentage  of  the  sample  that  identified  each  instance  correctly. 

•  Construct  a  frequency  distribution  of  the  examples  that  will  show  the  range  of 
difficulty  of  the  instances. 

5.  Prepare  a  Diagnostic  Classification  Test.  “A  classification  test  is  an  instrument  which 
enables  the  instructor  to  make  valid  and  reliable  inferences  about  the  student’s  ability  to 
classify  newly  encountered  instances  of  the  concept(s)”  (Merrill  &  Tennyson,  1977,  p.  79). 
Classification  behavior  is  best  measured  by  presenting  students  with  new  examples  and 
nonexamples  of  the  concept  class  and  having  them  identify  which  are  members  and  which 
are  nonmembers. 

6.  Prepare  Attribute  Isolation.  Use  an  attention  focusing  device  (e.g.,  color,  drawings, 
special  symbols,  simplified  illustrations,  etc.)  to  direct  the  student’s  attention  to  the  critical 
attributes  of  a  specific  example;  to  potentially  confusing  variable  attributes  in  examples  and 
nonexamples;  and  to  the  absence  of  the  critical  attributes  in  a  specific  nonexample. 

Caution:  don’t  use  attention  focusing  devices  merely  to  emphasize.  Be  sure  to  emphasize 
critical  attributes  rather  than  variable  attributes. 

7.  Prepare  an  Instructional  Strategy.  “An  instructional  strategy  is  a  specified  sequence  of 
presentation  forms”  (Merrill  &  Tennyson,  1977,  p.  111).  Ensure  that  the  instructional 
strategy  includes  all  five  of  the  components  described  above:  knowledge  structure, 
presentation,  exploration,  practice,  and  learner  guidance  (Merrill,  1997). 
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Recommendations  for  Teaching  Specific  Types  of  Content 

(Continued) 


Howard  (2000)  maintains  that  the  more  that  students  practice  classifying  varied  and  novel 
examples,  the  higher  the  probability  that  they  will  be  able  to  classify  new  examples  in  a  variety 
of  contexts. 

Teaching  Causal  Principles 

According  to  Merrill  (1983),  the  effective  instruction  of  a  cause-and-effect  principle  follows 
these  steps: 

1 .  Provide  a  statement  of  the  cause(s)  and  the  resulting  effect(s). 

2.  Next,  provide  an  example  (a  prototype  or  worked  example  of  an  application  of  the  causal 
principle  in  a  work  setting). 

3.  Follow  this  with  activities  that  allow  the  student  to  work  with  the  causal  principle.  These 
activities  should  stare  with  simple  examples  and  move  to  more  complex  examples.  The 
student  should  first  be  asked  to  describe  and  label  each  phase  of  the  cause-and-effect 
chain  in  the  correct  order.  Then,  when  given  one  phase,  he  or  she  should  be  asked  to 
predict  the  next  phase  or  the  previous  phase. 

4.  Finally,  the  student  should  be  asked  to  use  the  principle  to  solve  increasingly  difficult  or 
novel  problems. 

Teaching  Process  Knowledge 

Merrill  (2000)  defines  process  knowledge  as  “knowledge  about  how  something  works.  It 
answers  the  question,  ‘What  happens?’”  (p.  12).  Provide  a  clear  narrative  description  of  the 
process  integrated  with  a  visual  model  of  the  sequence  of  events  that  characterize  the  process. 
Describe  each  stage  of  the  process,  including  the  key  events  that  occur  at  each  stage  to  produce  a 
change  that  leads  to  the  next  stage. 

Teaching  Procedural  (“How  To”)  Knowledge 

Procedural  knowledge  may  include  a  combination  of  the  other  knowledge  types.  Instruction 
should  provide  clear  step-by-step  descriptions  of  all  actions  and  decisions  required  to  achieve  the 
end  goal.  It  should  also  include  models  or  demonstrations  of  the  procedure  and/or  worked 
examples  that  provide  knowledge  of  the  concepts,  processes,  and  principles  involved  in  the 
procedure.  The  student  should  be  given  the  opportunity  to  practice  the  procedure  on  a  variety  of 
problems  and  in  settings  that  are  similar  to  the  work  environment  where  the  procedure  will  be 
used. 


163 


Technical  Report  2005-002 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


164 


Technical  Report  2005-002 


Appendix  B 

Motivational  Strategies  According  to  the  ARCS  Model 

(Keller,  1987,  pp.  4-5) 

The  following  motivational  strategies  are  focused  on  classroom  instruction.  However,  they 
can  be  modified  for  other  types  of  instructional  delivery  (e.g.,  web  or  CD-ROM).  If  other 
instructional  delivery  methods  are  chosen,  it  is  essential  that  the  functions  of  the  live  instructor 
(e.g.,  providing  helpful  feedback)  be  included  as  part  of  the  content. 

Attention  Strategies 


Al:  Incongruity,  Conflict 

A 1 . 1  Introduce  a  fact  that  seems  to  contradict  the  learner’s  past  experience. 

A 1 .2  Present  an  example  that  does  not  seem  to  exemplify  a  given  concept. 

A 1 .3  Introduce  two  equally  plausible  facts  or  principles,  only  one  of  which  can  be  true. 

Al .4  Play  devil’s  advocate. 

A2:  Concreteness 

A2. 1  Show  visual  representations  of  any  important  object  or  set  of  ideas  or  relationships. 

A2.2  Give  examples  of  every  instructionally  important  concept  or  principle. 

A2.3  Use  content-related  anecdotes,  case  studies,  biographies,  etc. 

A3:  Variability 

A3. 1  In  stand  up  delivery,  vaiy  the  tone  of  your  voice,  and  use  body  movement,  pauses,  and  props. 

A3. 2  Vary  the  format  of  instruction  (information  presentation,  practice,  testing,  etc.)  according  to  the  attention 
span  of  the  audience. 

A3.3  Vary  the  medium  of  instruction  (platform  delivery,  film,  video,  print,  etc.). 

A3. 4  Break  up  print  materials  by  use  of  white  space,  visuals,  tables,  different  typefaces,  etc. 

A3.5  Change  the  style  of  presentation  (humorous-serious,  fast-slow,  loud-soft,  active-passive,  etc.). 

A3.6  Shift  between  student-instructor  interaction  and  student-student  interaction. 

A4:  Humor 

A4. 1  Where  appropriate,  use  plays  on  words  during  redundant  information  presentation. 

A4.2  Use  humorous  introductions. 

A4.3  Use  humorous  analogies  to  explain  and  summarize. 

A5:  Inquiry 

A5.1  Use  creativity  techniques  to  have  learners  create  unusual  analogies  and  associations  to  the  content. 

A5.2  Build  in  problem  solving  activities  at  regular  intervals. 

A5.3  Give  learners  the  opportunity  to  select  topics,  projects  and  assignments  that  appeal  to  their  curiosity  and 
need  to  explore. 

A6:  Participation 

A6. 1  Use  games,  role  plays,  or  simulations  that  require  learner  participation. 
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Motivational  Strategies  According  to  the  ARCS  Model 

(Continued) 

Relevance  Strategies 


Rl:  Experience 

R1 . 1  State  explicitly  how  the  instruction  builds  on  the  learner’s  existing  skills. 

Rl  .2  Use  analogies  familiar  to  the  learner  from  past  experience. 

Rl  .3  Find  out  what  the  learners’  interests  are  and  relate  them  to  the  instruction. 

R2:  Present  Worth 

R2. 1  State  explicitly  the  present  intrinsic  value  of  learning  the  content,  as  distinct  from  its  value  as  a  link  to 
future  goals. 

R3:  Future  usefulness 

R3. 1  State  explicitly  how  the  instruction  relates  to  future  activities  of  the  learner. 

R3.2  Ask  learners  to  relate  the  instruction  to  their  own  future  goals  (future  wheel). 

R4:  Need  Matching 

R4. 1  To  enhance  achievement  striving  behavior,  provide  opportunities  to  achieve  standards  of  excellence  under 
conditions  of  moderate  risk. 

R4.2  To  make  instruction  responsive  to  the  power  motive,  provide  opportunities  for  responsibility,  authority, 
and  interpersonal  interaction. 

R4.3  To  satisfy  the  need  for  affiliation,  establish  trust  and  provide  opportunities  for  no-risk,  cooperative 
interaction. 

R5:  Modeling 

R5. 1  Bring  in  alumni  of  the  course  as  enthusiastic  guest  lecturers. 

R5.2  In  a  self-paced  course,  use  those  who  finish  first  as  deputy  tutors. 

R5.3  Model  enthusiasm  for  the  subject  taught. 

R6:  Choice 

R6. 1  Provide  meaningful  alternative  methods  for  accomplishing  a  goal. 

R6.2  Provide  personal  choices  for  organizing  one’s  work. 


Confidence  Strategies 

Cl:  Learning  Requirements 

C 1 . 1  Incorporate  clearly  stated,  appealing  learning  goals  into  instructional  materials. 

Cl .2  Provide  self-evaluation  tools  which  are  based  on  clearly  stated  goals. 

C 1 .3  Explain  the  criteria  for  evaluation  of  performance. 

C2:  Difficulty 

C2. 1  Organize  materials  on  an  increasing  level  of  difficulty;  that  is,  structure  the  learning  material  to  provide  a 
“conquerable”  challenge. 
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Motivational  Strategies  According  to  the  ARCS  Model 

(Continued) 


C3:  Expectations 

C3.1  Include  statements  about  the  likelihood  of  success  with  given  amounts  of  effort  and  ability. 

C3.2  Teach  students  how  to  develop  a  plan  of  work  that  will  result  in  goal  accomplishment. 

C3.3  Help  students  set  realistic  goals. 

C4:  Attributions 

C4.1  Attribute  student  success  to  effort  rather  than  luck  or  ease  of  task  when  appropriate  (i.e.,  when  you  know 
it’s  true!). 

C4.2  Encourage  student  efforts  to  verbalize  appropriate  attributes  for  both  successes  and  failures. 

C5:  Self-Confidence 

C5. 1  Allow  students  opportunity  to  become  increasingly  independent  in  learning  and  practicing  a  skill. 

C5.2  Have  students  learn  new  skills  under  low  risk  conditions,  but  practice  performance  of  well-learned  tasks 
under  realistic  conditions. 

C5.3  Help  students  understand  that  the  pursuit  of  excellence  does  not  mean  that  anything  short  of  perfection  is 
failure;  learn  to  feel  good  about  genuine  accomplishment. 


Satisfaction  Strategies 


SI:  Natural  Consequences 

5 1 . 1  Allow  a  student  to  use  a  newly  acquired  skill  in  a  realistic  setting  as  soon  as  possible. 

5 1 .2  Verbally  reinforce  a  student’s  intrinsic  pride  in  accomplishing  a  difficult  task. 

5 1 .3  Allow  a  student  who  masters  a  task  to  help  others  who  have  not  yet  done  so. 

S2:  Unexpected  Rewards 

52. 1  Reward  intrinsically  interesting  task  performance  with  unexpected,  non-contingent  rewards. 

52.2  Reward  boring  tasks  with  extrinsic,  anticipated  rewards. 

S3:  Positive  Outcomes 

53. 1  Give  verbal  praise  for  successful  progress  or  accomplishment. 

53.2  Give  personal  attention  to  students. 

53.3  Provide  informative,  helpful  feedback  when  it  is  immediately  useful. 

53.4  Provide  motivating  feedback  (praise)  immediately  following  task  performance. 

S4:  Negative  Influences 

54. 1  Avoid  the  use  of  threats  as  a  means  of  obtaining  task  performance. 

54.2  Avoid  surveillance  (as  opposed  to  positive  attention). 

54.3  Avoid  external  performance  evaluations  whenever  it  is  possible  to  help  the  student  evaluate  his  or  her  own 
work. 

S5:  Scheduling 

55. 1  Provide  frequent  reinforcements  when  a  student  is  learning  a  new  task. 

55.2  Provide  intermittent  reinforcement  as  a  student  becomes  more  competent  at  a  task. 

55.3  Vary  the  schedule  of  reinforcements  in  terms  of  both  interval  and  quantity. 
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Job  Aid  for  Selecting  Action  Verbs 

(Department  of  Defense,  2001,  p.  50-52) 


KNOWLEDGE 

Verbs 

Learning  Level  and  Definition 

Advise 

Elaborate 

List 

Recount 

Learning  Level:  Fact  Learning  Definition: 

Answer 

Express 

Name 

Specify 

Verbal  or  symbolic  information  (e.g., 

Brief 

Identify 

Read 

State 

names,  formulas,  facts,  etc.). 

Calculate 

Inform 

Recall 

Tell 

Define 

Instruct 

Recommend 

Appraise 

Compute 

Evaluate 

Measure 

Learning  Level:  Rule  Learning 

Compile 

Encrypt 

Format 

Outline 

Definition:  Using  two  or  more  facts  in  a 

Compose 

Estimate 

Forward 

Route 

manner  that  provides  regularity  of  behavior  in 

an  infinite  variation  of  situations. 

Check 

Delete 

Pause 

Start 

Learning  Level:  Procedure  Learning 

Condense 

Implement 

Resume 

Stop 

Definition:  Performing  step-by-step  actions  in 

Edit 

Initiate 

Set  up 

the  proper  sequence 

Allocate 

Correlate 

Finalize 

'  Reorganize 

Learning  Level:  Discrimination  Learning 

Arrange 

Cross-check 

Group 

Restate 

Definition:  Grouping  similar  and  dissimilar 

Assign 

Designate 

Label 

Schedule 

items  according  to  their  distinct  characteristics. 

Categorize 

Differentiate 

Level 

Select 

Classify 

Discriminate 

Match 

Separate 

Collate 

Distinguish 

Organize 

Sort 

Compare 

Distribute 

Rank 

Task 

Confirm 

Divide 

Realign 

Template 

Consolidate 

Eliminate 

Redistribute 

Translate 

Contrast 

Extract 

Reexamine 

Tune 

Analyze 

Derive 

Generate 

Project 

Learning  Level:  Problem  Solving 

Annotate 

Design 

Hypothesize 

Resolve 

Definition:  Synthesizing  lower  levels  of 

Apply 

Determine 

Illustrate 

Revise 

knowledge  for  the  resolution  of  problems. 

Change 

Diagram 

Infer 

Search 

Combine 

Discover 

Investigate 

Solve 

Conclude 

Draft 

Locate 

Summarize 

Convert 

Effect 

Manipulate 

Synthesize 

Create 

Explain 

Modify 

Triage 

Criticize 

Extend 

Plan 

Use 

Decide 

Find 

Predict 

War  game 

Defend 

Generalize 

Produce 

SKILLS 

Verbs 

Learning  Levels  and  Definition 

Detect 

Hear 

Taste 

Learning  Level:  Perception  (Encoding) 

Feel 

Scan 

Visualize 

Definition:  Sensory  stimuli  that  translate  into 

mmSSM 

physical  performance. 

Assault 

Fall 

Learning  Level:  Gross  Motor  Skills 

Cany 

Hold 

Definition:  Manual  dexterity  in  the 

Creep 

Jump 

performance  of  physical  skills. 

Depart 

List 

Advance 

Guide 

Maneuver 

Take  off 

Learning  Level:  Continuous  Movement 

Control 

Hover 

Regulate 

Track 

Definition:  Tracking  or  making  compensatory 

Follow 

Land 

Steer 

Traverse 

movements  based  on  feedback. 

Able 

Cross 

Prepare 

Set 

Learning  Level:  Readiness 

Assist 

Delay 

Prime 

Stand  to 

Definition:  Having  readiness  to  take  a 

Challenge 

Guard 

Ready 

particular  action. 
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Job  Aid  for  Selecting  Action  Verbs 

(Continued) 


SKILLS  -  continued 


Verbs 

Learning  Level  and  Definition 

Access 

Dispose 

Mount 

Rotate 

Learning  Level:  Mechanism 

Activate 

Disseminate  Move 

Save 

Definition:  Performing  a  complex  physical  or 

Actuate 

Drive 

Navigate 

Secure 

mental  skill. 

Adjust 

Egress 

Obtain 

Send 

Administer 

Elevate 

Open 

Service 

Align 

Emplace 

Operate 

Shut  down 

Archive 

Employ 

Order 

Sight 

Arm 

Engage 

Park 

Signal 

Assemble 

Energize 

Perform 

Splint 

Balance 

Enter 

Place 

Squeeze 

Breach 

Establish 

Plot 

Stockpile 

Calibrate 

Evacuate 

Police 

Store 

Camouflage 

Exchange 

Position 

Stow 

Center 

Fill  out 

Post 

Strike 

Charge 

Fire 

Press 

Submit 

Clean 

Fit 

Pressurize 

Supervise 

Clear 

Fuel 

Process 

Support 

Close 

Ground 

Procure 

Sweep 

Collect 

Harden 

Provide 

Take 

Connect 

Hoist 

Publish 

Take  charge 

Cover 

Initialize 

Raise 

Tap 

Debrief 

Input 

Range 

Test 

Debug 

Insert 

Reach 

Tighten 

Decontaminate 

Inspect 

Receive 

Trace 

Deliver 

Install 

Record 

Transfer 

Destroy 

Integrate 

Reestablish 

Transmit 

Diagnose 

Intercept 

Refuel 

Transport 

Dig 

Isolate 

Release 

Treat 

Disassemble 

Issue 

Relocate 

Troubleshoot 

Disconnect 

Jack 

Remove 

Type 

Disengage 

Launch 

Repair 

Unload 

Dismantle 

Load 

Replace 

Update 

Dispatch 

Log 

Replenish 

Utilize 

Displace 

Lubricate 

Reset 

Write 

Display 

Maintain 

Manage 

Retrieve 

Return 

Zero 

Acclimatize 

Direct 

Occupy 

Reconcile 

Learning  Level:  Adaptation 

Accommodate 

Draw 

Orient 

Recover 

Definition:  Modifying  a  complex  physical  or 

Adapt 

Evade 

Pack 

Reduce 

mental  skill  to  accommodate  a  new  situation. 

Ambush 

Infiltrate 

Patrol 

Relieve 

Attach 

Lay 

Prevent 

Suppress 

Bypass 

Lead 

Program 

Tailor 

Conduct 

Map 

Protect 

Temper 

Deploy 

Neutralize 

Queue 

Train 

Cause 

Contrive 

Initiate 

Make 

Learning  Level:  Origination 

Construct 

Correct 

Invent 

Originate 

Definition:  Creating  a  new  complex  physical  or 
mental  skill  to  accommodate  a  new  situation. 
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(Continued) 


ATTITUDES 

Verbs 

Learning  Level  and  Definition 

Attend 

Monitor 

Reconnoiter 

Show 

Learning  Level:  Receiving 

closely 

Observe 

Recognize 

sensitivity 

(Perception;  Situation  Awareness) 

Listen 

Listen 

attentively 

Perceive 

Show 

awareness 

Wait 

Definition:  Demonstrating  mental  preparedness  to 
perceive  the  normal,  abnormal,  and  emergency  condition 
cues  associated  with  the  performance  of  an  operational 
procedure. 

Accomplish 

Complete 

Give 

Request 

Learning  Level:  Responding  (Interpreting) 

Achieve 

assignment 

Indicate 

Respond 

Definition:  Demonstrating  mental  preparedness  to  encode 

Acknowledge 

Comply 

Interpret 

Resume 

operational  cues  as  indicators  of  normal,  abnormal,  and 

Announce 

Demonstrate 

Notify 

Show 

emergency  conditions  associated  with  the  performance  of 

Ask 

Communicate 

Complete 

Describe 

Encode 

Execute 

Obey  rules 
React 

Report 

an  operational  procedure. 

Alert 

Belief 

Prioritize 

Share 

Learning  Level:  Valuing  (Judgment) 

Appreciate 

Cancel 

Propose 

Study 

Definition:  Demonstrating  the  ability  to  judge  the  worth 

Approve 

Choose 

Qualify 

Validate 

or  quality  of  normal,  abnormal,  and  emergency  cues 

Assess 

Judge 

Reassess 

Verify 

associated  with  the  performance  of  an  operational 

Authenticate 

Justify 

Review 

procedure. 

Allow 

Command 

Ensure 

Serve 

Learning  Level:  Competence  (Application  of  resource 

Alter 

Coordinate 

Influence 

management  strategies  and  tactics.) 

Assume 

Enforce 

Prescribe 

Definition:  Demonstrating  the  mental  preparedness  to 
make  decisions  using  prioritized  strategies  and  tactics  in 
response  to  normal,  abnormal,  and  emergency  condition 
cues  associated  with  the  performance  of  operational 
procedures. 

Conceive 

Formulate 

Innovate 

Learning  Level:  Innovation 

Conjecture 

Devise 

Imagine 

(Generation  of  new  resource  management  strategies  and 
tactics) 

Definition:  Demonstrating  the  mental  preparedness  to 
make  decisions  by  generating  the  results  expected  upon 
completion  of  prioritized  strategies  or  tactics  in  response 
to  normal,  abnormal,  and  emergency  cues  associated  with 
performance  of  an  operational  procedure,  and  generating 
prioritized  strategies  and  tactics  in  response  to  abnormal 
or  emergency  cues. 
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Interactivity  Assessment  Scales 

(Adapted  from:  Roblyer  &  Wiencke,  2003) 


Instructions:  If  you  are  evaluating  an  instructor-mediated  course,  you  may  wish  to  use  the 
following  scales.  Read  the  statements  below  each  scale  and  circle  the  number  that  best 
characterized  the  course  on  each  rating  element.  Sum  the  scores  to  give  an  overall  summary  of 
interactivity  (higher  scores  indicate  a  better  level  of  interactivity).  Scores  on  each  rating  element 
can  help  diagnose  specific  interactivity  problems. 


Element  1.  Social/Rapport-Building  Designs  for  Interaction 


1  2  3  4  5 


Instructor  does  not 
encourage  students 
to  get  to  know  one 
another  on  a 
personal  basis. 
Activities  do  not 
require  social 
interaction. 


Instructor  requires: 

•  brief 
introductions 

•  one  exchange  of 
personal 

information  (e.g., 
written  bio  of 
background  & 
experiences). 


Instructor  requires: 

•  all  #2 
and 

•  one  in-class 
activity  designed 
to  increase 
communication 


Instructor  requires: 

•  all  #3 

and 

Instructor  also 
interacts  with 
students  on  a 
social/personal 
basis. 


Instructor  requires: 

•  all  #4 

and 

Instructor  provides 
ongoing  course 
structure  to 
promote  social 
rapport  among 
students  and 
instructor. 


Element  2.  Instructional  Designs  for  Interaction 


1  2  3  4  5 


Only  one-way 
delivery  of 
information  (e.g., 
lectures  or  text 
delivery)  and 
student  products 
based  on 
information 


Instructional 

activities: 

•  only  student- 
instructor 
communication 
(asking/responding 
to  questions) 


Instructional 

activities: 

•  #2 
and 

•  student  -student 
communication 
(e.g.  discussion 
pairs  or  small 
groups) 


Instructional 

activities: 

•  #3 
and 

•  students  develop 
products 

cooperatively  (e.g., 
in  pairs  or  small 
groups). 


Instructional 

activities: 

•  #4 
and 

•  groups  share 
feedback  with 
other  groups 
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Interactivity  Assessment  Scales 

(Continued) 


Element  3.  Interactivity  of  Technology  Resources 


1 

2 

3 

4 

5 

Fax,  Web  pages 

•  E-mail,  Listserv, 

Technologies: 

Technologies: 

Technologies: 

(text  and/or 

conference/bulletin 

•  all  #2 

•  all  #4 

•  all  #5 

graphics). 

board  (2-way 
asynchronous 
exchange  of  info.) 

and 

•  chat  room  or 
other  technologies 
(synchronous 
exchange  of  info., 
primarily  written) 

and 

•  1  -way  visuals 
(e.g., 

teleconferencing) 
and  2-way  voice . 

and 

•  visual 

technologies  that 
allow  2-way 
synchronous  visual 
and  voice 
(instructor-student 
&  student-student) 

Element  4.  Evidence  of  Learner  Engagement 


| 

1 

1 

I 

1 

1 

2 

3 

4 

5 

By  end  of  course, 

By  end  of  course, 

By  end  of  course, 

By  end  of  course, 

By  end  of  course, 

most  students 

most  students 

all  or  nearly  all 

most  students 

all  or  nearly  all 

(50%-75%)  are 

(50%-75%)  are 

students  (90%- 

(50%-75%)  are 

students  (90%- 

replying  to 

replying  to 

100%)  are  replying 

both  replying  to 

100%)  are  both 

messages  from 

messages  from 

to  messages  from 

and  initiating 

replying  to  and 

instructor  but  only 

instructor  and 

instructor  and 

messages  fboth 

initiating  messages 

when  required; 

other  students, 

other  students 

required  and 

(both  required  and 

messages  are 

both  when 

(both  required  & 

voluntary); 

voluntary); 

sometimes 

required  and  on  a 

voluntary);  replies 

messages  are 

messages  are 

unresponsive  to 

voluntary  basis; 

are  alwavs 

detailed  and 

detailed, 

topics  and  tend  to 

replies  are  usually 

resDonsive  to 

responsive  to 

responsive  to 

be  either  brief  or 

responsive  to 

topics  but 

tQpics  and  usually 

topics,  and  are 

wordy  &  rambling. 

topics  but  are 
either  brief  or 
wordy  &  rambling. 

sometimes  are 
either  brief  or 
wordy  &  rambling. 

reflect  and  effort  to 
communicate  well. 

well-developed 

communications. 

174 


Technical  Report  2005-002 


Appendix  D 

Interactivity  Assessment  Scales 

(Continued) 


Element  5.  Evidence  of  Instructor  Engagement 

I _ I _ I _ I _ I 

1  2  3  4  5 


Instructor  responds  Instructor  responds  Instructor  responds  Instructor  responds  Instructor  responds 

only  randomly  to  to  most  student  to  aU  student  to  aH  student  to  ah  student 

student  queries;  queries;  responses  queries;  responses  queries;  responses  queries;  responses 

responses  usually  usually  are  within  usually  are  within  usually  are  prompt  are  always  prompt 

take  more  than  48  48  hours;  feedback  48  hours;  feedback  (i.e.,  within  24  (i.e.,  within  24 

hours;  feedback  is  sometimes  offers  usually  offers  hours);  feedback  hours);  feedback 

brief  and  provides  some  analysis  of  some  analysis  of  always  offers  always  offers 

little  analysis  of  student  work  and  student  work  and  detailed  analysis  of  detailed  analysis  of 

student  work  or  suggestions  for  suggestions  for  student  work  and  student  work  and 

suggestions  for  improvement.  improvement.  suggestions  for  suggestions  for 

improvement.  improvement.  improvement 

along  with 
additional  hints 
and  information  to 
supplement 


learning. 
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